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ABSTRACT 


Plant  procedures  are  instructions  to  guide  operators  in  monitoring,  decision  making,  and  controlling  nuclear  power 
plants.  While  plant  procedures  historically  have  been  paper-based,  computer-based  procedures  (CBPs)  are  being 
developed  to  support  procedure  use.  CBPs  have  a  range  of  capabilities  that  may  support  operators  and  reduce 
demands  associated  with  paper  procedures.  The  objective  of  this  study  was  to  establish  human  factors  review 
guidance  for  CBP  systems  based  on  a  technically  valid  methodology.  While  the  study  mainly  addressed  emergency 
operating  procedures,  much  of  the  guidance  developed  applies  to  other  types  of  procedures.  First,  a  CBP 
characterization  was  developed  for  describing  their  key  design  features  including  both  procedure  representation 
and  functionality.  Then,  the  research  on  CBPs  and  related  areas  was  reviewed.  This  information  provided  the 
technical  basis  on  which  the  guidelines  for  design  review  were  developed.  The  review  guidelines  address  both  the 
design  process  and  the  implementation  of  CBP  systems.  For  some  aspects  of  CBPs  the  technical  basis  was 
insufficient  to  develop  guidance;  these  aspects  were  identified  as  issues  to  be  addressed  in  future  research. 
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EXECUTIVE  SUMMARY 


The  Human-System  Interface  Design  Review  Guideline,  NUREG-0700,  Rev.  1  (O'Hara  et  al.,  1996),  was 
developed  to  provide  guidance  on  human  factors  engineering  (HFE)  for  the  U.S.  Nuclear  Regulatory  Commission 
(NRC).  The  NRC  staff  uses  NUREG-0700  for  (1)  reviewing  submittals  of  human-system  interface  (HSI)  designs 
prepared  by  licensees  or  applicants  for  a  license  or  design  certification  of  a  commercial  nuclear  power  plant  (NPP), 
and  (2)  undertaking  HSI  reviews  that  could  be  included  in  an  inspection  or  other  types  of  regulatory  review  of  HSI 
designs,  or  incidents  involving  human  performance.  It  describes  those  aspects  of  the  HSI  design  review  process 
that  are  important  to  identifying  and  resolving  human  engineering  discrepancies  that  could  adversely  affect  plant 
safety.  NUREG-0700  also  has  detailed  HFE  guidelines  for  assessing  the  implementation  of  HSI  designs. 

In  generating  NUREG-0700,  Rev.  1,  several  topics  were  identified  as  “gaps”  because  there  was  an  insufficient 
technical  basis  upon  which  to  formulate  guidance.  One  such  topic  is  the  integration  of  advanced  HSI  technology 
into  conventional  NPPs.  The  NRC  is  currently  sponsoring  research  at  Brookhaven  National  Laboratory  (BNL)  to 
(1)  better  define  the  effects  of  changes  in  HSIs  brought  about  by  incorporating  digital  technology  on  personnel 
performance  and  plant  safety,  and  (2)  develop  HFE  guidance  to  supjxnt  safety  reviews,  should  a  review  of  plant 
modifications  or  HSIs  be  necessary. 

Based  upon  the  literature,  interviews,  and  site  visits,  O'Hara  et  al.  (1996)  identified  changes  in  HSI  technology  and 
their  potential  effects  on  personnel  performance.  The  topics  were  then  evaluated  for  their  potential  safety 
significance  (Stubler  et  al.,  1996);  computer-based  procedures  (CBPs)  was  one  HSI  technology  that  was  found  to  be 
potentially  safety  significant.  (The  safety  analysis  is  described  in  more  detail  in  Section  5.4.2.2  of  this  report.) 

Plant  procedures  provide  instructions  to  guide  operators  in  monitoring,  decision  making,  and  controlling  the  plant. 
Historically,  plant  procedures  have  been  paper-based  and  were  not  considered  part  of  the  HSI.  Following  the 
accident  at  Three  Mile  Island,  the  nuclear  power  industry  recognized  the  importance  of  having  technologically 
sound  and  easy-to-use  procedures  to  handle  major  plant  disturbances.  For  emergency  operations,  symptom-based 
procedures  were  established  that  enabled  operating  crews  to  restore  and  maintain  the  plant’s  safety  functions 
without  having  to  diagnose  events  or  the  specific  causes  of  process  disturbances. 

Paper-based  procedures  (PBPs)  have  characteristics  that  limit  how  information  can  be  presented  to  the  operators. 
These  limitations  include  presenting  information  in  sequential  form,  requiring  numerous  iterations  through  steps, 
and  cautions  or  warnings  that  may  not  be  applicable  for  all  system  states  (Wourms  and  Rankin,  1994;  Mampaey  et 
al.,  1988).  PBPs  also  impose  tasks  on  the  operator  that  are  not  directly  related  to  controlling  the  plant.  To  make 
transitions  between  procedure  steps  and  documents,  and  maintain  awareness  of  the  status  of  procedures  that  are  in 
progress,  operators  must  handle,  arrange,  scan,  and  read  PBPs  in  parallel  with  monitoring  and  control  tasks. 

CBPs  are  being  developed  to  support  procedure  management.  CBPs  have  a  range  of  capabilities  that  may  support 
operators  in  controlling  the  plant  and  reduce  the  demands  associated  with  PBPs.  In  their  simplest  form,  CBPs 
show  the  same  information  via  computer-driven  video  display  units  (VDUs).  More  advanced  CBPs  may  include 
features  to  support  managing  procedures  (e.g.,  making  transitions  between  steps  emd  documents,  and  maintaining 
awareness  of  procedures  in  progress),  detecting  and  monitoring  the  plant’s  state  and  parameters,  interpreting  its 
status,  and  selecting  actions  and  executing  them. 

The  objective  of  this  study  was  to  develop  HFE  review  guidance  for  CBP  systems  based  on  a  technically  valid 
methodology.  To  support  this  objective,  the  following  tasks  were  undertaken: 

•  Development  of  a  framework  for  characterizing  key  design  features  of  CBP  systems 

•  Development  of  a  technical  basis  using  research  and  analyses  on  human  performance  relevant  to  CBPs 
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•  Development  of  HFE  review  guidelines  for  CBPs  in  a  format  that  is  consistent  with  NUREG-0700,  Rev.  1,  and 
NUREG-0711 

•  Identification  of  remaining  CBP  issues  for  which  research  was  insufficient  to  support  our  development  of  NRC 
review  guidance 

The  status  of  each  will  be  briefly  addressed  below. 

CBP  System  Characterization  Framework 

For  this  study,  CBP  systems  were  narrowly  defined  to  encompass  computer  systems  that  support  procedure 
presentation  and  use.  The  focus  was  on  the  HFE  aspects  of  CBPs,  and  not  the  I&C  or  software  aspects  (although 
the  latter  are  important  as  well,  and  are  described  in  other  NRC  regulatory  and  research  programs).  CBPs  were 
characterized  along  the  following  dimensions: 

•  Representation  of  Procedure  Elements 

•  Procedure  Functionality 

•  Interface  Management  and  Support 

•  CBP  Hardware 

•  Backup  Systems  for  Procedures 

•  Integration  of  CBP  System  with  the  HSI 
Development  of  the  Technical  Basis 

The  effects  of  CBPs  on  crew  performance  were  determined  by  examining  three  types  of  research:  (1)  empirical 
studies  of  CBPs  where  data  on  personnel  performance  were  collected,  (2)  analyses  of  personnel  performance  using 
models,  and  (3)  expert  opinion  about  their  postulated  effects  on  personnel  performance. 

The  human  performance  research  was  organized  into  three  categories:  comparisons  of  CBP  and  PBP  systems, 
observations  of  operators’  use  of  CBPs,  and  comparisons  of  design  characteristics  of  procedures.  Several 
conclusions  were  made  from  comparing  CBPs  with  PBPs: 

•  Operators  perform  tasks  more  quickly. 

•  Operators’  overall  cognitive  workload  is  reduced. 

•  Operators  may  make  fewer  errors  in  transitioning  through  procedures. 

•  Operators  may  accept  CBPs  readily  and  find  them  easier  to  use. 

However,  much  of  the  human  performance  research  had  insufficient  detail  to  evaluate  its  generalizability.  Studies 
that  were  sufficiently  documented  had  potential  methodological  weaknesses  which  limited  their  conclusiveness  and 
generalization. 
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Personnel  performance  was  analyzed  with  two  classes  of  techniques:  performance  models  and  risk  models.  The 
performance  models  showed  no  clear  advantage  of  CBPs  over  PBPs.  Instead,  they  illustrated  the  imp>ortance  of 
performance  tradeoffs  in  assessing  different  procedure  systems.  In  general,  complexity  and  attentional  demands 
were  higher,  while  data  retrieval  was  easier  and  task  completion  time  was  less  for  CBPs.  Similarly,  mixed  results 
were  obtained  from  the  risk  analyses.  They  illustrated  the  potential  for  these  systems  to  improve  performance  by 
supporting  such  procedure-related  activities  as  process  monitoring,  logic  analysis,  navigation,  and  place  keeping. 
However,  when  poorly  implemented,  CBPs  can  reduce  human  reliability. 

Finally,  the  SME  review  of  CBPs  identified  many  positive  aspects  of  their  use  on  the  crew’s  performance. 

However,  they  also  identified  a  wide  range  of  issues  to  be  resolved  in  developing  CBPs.  The  review  highlighted 
the  importance  of  considering  HFE  activities  in  CBP  development,  e.g.,  the  integration  of  the  CBP  system  with  the 
other  HSls  and  with  the  overall  operational  philosophy  of  the  plant.  Thorough  V&V  programs  were  also 
emphasized.  In  general,  these  findings  were  consistent  with  the  information  discussed  earlier. 

When  considering  all  the  results,  we  concluded  that  there  is  evidence  that  CBPs  can  support  and  enhance  operator 
performance.  However,  important  issues  remain  to  be  addressed  both  in  research  and  in  the  development  of 
individual  systems.  Thus,  we  repeat  the  advice  of  researchers  and  developers:  CBP  systems  should  be  developed  in 
such  a  way  that  their  benefits  and  drawbacks  can  be  fully  evaluated  for  each  specific  system.  CBPs  have  important 
impacts  on  NPP  operations,  some  of  which  extend  beyond  those  the  designers  intended. 

Reflecting  this  approach,  we  offer  some  general  considerations  for  near-term  approaches  to  CBP  systems: 

•  Support  cognitive  functions  that  may  be  distracting  and  error  prone,  such  as 

-  process  monitoring 

-  logic  analysis  (cautiously  so  not  to  underspecify  the  analysis  and  undermine  operator’s  judgement) 

•  Support  procedure  management,  e.g.,  step  completion,  place  keeping,  transitioning  between  procedures 

•  Provide  PBP  backup  systems  and  ensure  similarity  of  CBPs  and  PBPs  in  order  to  (1 )  ensure  confidence  in 
near-term  CBP  applications,  (2)  enable  operating  experience  to  be  gained,  (3)  minimize  the  impact  on 
function  allocation,  (4)  ease  the  training  burdens  associated  with  both  systems,  and  (5)  ensure  successful  crew 
performance  when  transitions  to  and  from  backups  are  necessary  (minimize  the  potential  for  negative  transfer 
or  difficulties  in  performance) 

HFE  Review  Guidelines 


Guidance  for  the  review  of  CBPs  was  developed  to  address  the  CBP  design  process  and  HFE  design.  Both  types  of 
guidance  are  needed  for  a  design  review.  That  is,  while  there  was  a  sufficient  technical  basis  to  develop  detailed 
guidance  for  design-implementation  review,  as  is  typical  in  NUREG-0700,  several  limitations  in  the  technical  basis 
were  identified.  Many  issues  (listed  below)  remain  for  which  typical  NUREG-0700  guidance  could  not  be 
developed.  Therefore,  until  the  additional  guidance  is  developed,  these  issues  should  be  addressed  for  specific  CBP 
systems  using  CBP  design  process  guidance. 
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CBP  Issues 


As  noted  above,  several  human  performance  issues  associated  with  CBPs  were  identified.  They  represent  topics  for 
which  research  is  necessary  before  developing  guidance.  From  a  regulatory  review  perspective,  many  of  them  can 
be  dealt  with  on  a  case-by-case  basis  during  the  design  process  review.  Briefly,  the  issues  included  the  following: 

Methodological  and  Criterion  Requirements  for  Evaluating  CBP  Effects  -  Most  of  the  studies  reviewed  had 
methodological  weaknesses  which  limited  their  conclusiveness  and  generalizability.  This  issue  addresses  the  need 
to  evaluate  CBPs  and  their  effects  on  crew  performance  comprehensively,  to  better  understand  them  under  a  wide 
range  of  scenarios  and  complex  situations,  using  varied  measures  of  personnel  and  system  performance. 

Role  of  Plant  Personnel  in  Procedure  Management  -  This  issue  addresses  the  need  to  determine  how  to  design  and 
review  CBP  systems  (1)  to  allow  operators  to  maintain  an  independent  perspective  and  to  recognize  the 
procedure’s  contribution  to  higher-level  safety  goals,  (2)  to  automate  distracting  and  lower-level  error-prone  tasks, 
and  (3)  to  monitor  the  crew’s  performance,  especially  when  the  crew  and  CBPs  disagree. 

Team  Performance  -  This  issue  addresses  the  requirement  to  explore  the  effect  of  CBPs  on  crew  member’s  roles, 
teamwork,  and  communication.  How  CBPs  can  be  designed  to  effectively  promote  both  is  considered  as  well. 

Situation  Awareness,  Response  Planning,  and  Operator  Error  -  This  issue  addresses  the  need  to  assess  the  effect  of 
CBPs  on  situation  awareness  including: 

•  Procedure  management,  such  as  status  of  procedure  steps,  how  procedures  are  structured,  and  the  current 
location  within  a  procedure  or  between  a  set  of  procedures 

•  The  appropriateness  of  procedures  for  achieving  high-level  procedure  goals 

•  The  plant’s  status 

Level  of  Automation  of  Procedure  Functions  -  This  issue  addresses  the  need  to  evaluate  the  tradeoffs  between 
automating  procedure  functions,  e.g.,  the  analysis  of  procedure  step  logic,  and  the  operator’s  involvement, 
independence,  and  supervisory  control. 

Keyhole  Effects  and  Use  of  Multiple  CBP  Procedures  -  This  issue  concerns  the  requirement  to  evaluate  the 
significance  of  the  keyhole  effect  in  situations  where  operators  are  required  to  be  in  multiple  procedures  and  must 
access  information  in  parallel. 

CBP  Failure  in  Complex  Situations  -  This  issue  involves  the  need  to  evaluate  operator’s  management  of  the 
transition  from  CBPs  to  PBPs  and  back  to  CBPs  under  complex  conditions,  e.g.,  in  a  situation  where  operators  are 
deep  into  the  procedures,  multiple  procedures  are  open,  many  steps  are  completed,  many  are  continuously 
applicable,  and  time  and  parameter  steps  are  being  monitored  by  the  CBPs. 

Hybrid  Procedure  Systems  -  This  issue  addresses  the  need  to  evaluate  any  differential  effects  of  having  all  plant 
procedures  presented  in  a  CBP  system  versus  a  hybrid  system,  e.g.,  EOPs  presented  using  CBPs  and  all  other 
procedures  are  paper-based. 

Specific  CBP  Design  Features  -  This  issue  addresses  the  need  to  evaluate  the  relative  effects  of  specific  CBP 
design  features  on  performance. 
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BNL 

Brookhaven  National  Laboratory 

BWR 

Boiling  water  reactor 

CALS 

Continuous  acquisition  and  life-cycle  support 

CANDU 

Canadian  Deuterium  Uranium  Reactor 

CBP 

Computer-based  procedure 

CFR 

Code  of  Federal  Regulations  (U.S.) 

COMPRO 

Westinghouse  Computerized  Procedure 

COPMA 

Computerized  Operation  Manual 

COPRO 

Computerized  Procedure 

COSS 

Computerized  operator  support  system 

CR 

Control  room 

CSF 

Critical  safety  function 

DDD 

Detection-diagnosis-decision  making 

DOD 

Department  of  Defense  (U.S.) 

DOE 

Department  of  Energy  (U.S.) 

DSIN 

Nuclear  Installations  Safety  Directorate  (France) 

EdF 

Electricite  de  France 

EOP 

Emergency  operating  procedure 

EOPTS 

Emergency  Operating  Procedure  Tracking  System 

EPRI 

Electric  Power  Research  Institute 

GE 

General  Electric 
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General  technical  guidance 
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Human  factors  engineering 

HRA 

Human  reliability  analysis 

HSl 

Human-system  interface 

I&C 

Instrumentation  and  control 

IAEA 

International  Atomic  Energy  Association 
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Institute  for  Nuclear  Safety  and  Protection  (France) 

lETM 

Interactive  Electronic  Technical  Manuals 

ISLOCA 
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Knowledge-based  system 
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PRA 

Probabilistic  risk  assessment 
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RO 
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SDT 
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Subject-matter  expert 

SPDS 
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Three  Mile  Island  (nuclear  power  plant) 

URD 

Utility  Requirements  Document 

V&V 

Verification  and  validation 

VDU 

Video  display  unit 
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1  INTRODUCTION 


1.1  Background 

The  Human-System  Interface  Design  Review  Guideline,  NUREG-0700,  Rev.  1,  (O'Hara  et  al.,  1996)  was 
developed  to  provide  guidance  on  human  factors  engineering  (HFE)  for  the  U.S.  Nuclear  Regulatory  Commission 
(NRC).  The  NRC  staff  uses  NUREG-0700  for  (1)  reviewing  submittals  of  human-system  interface  (HSl)  designs 
prepared  by  licensees  or  applicants  for  a  license  or  design  certification  of  a  commercial  nuclear  power  plant  (NPP), 
and  (2)  undertaking  HSl  reviews  that  could  be  included  in  an  inspection  or  other  types  of  regulatory  review  of  HSl 
designs,  or  incidents  involving  human  performance.  It  describes  those  aspects  of  the  HSl  design  review  process 
that  are  important  to  identifying  and  resolving  human  engineering  discrepancies  that  could  adversely  affect  plant 
safety.  NUREG-0700  also  has  detailed  HFE  guidelines  for  assessing  the  implementation  of  HSl  designs. 

In  generating  NUREG-0700,  Rev.  1,  several  topics  were  identified  as  “gaps”  because  there  was  an  insufficient 
technical  basis  upon  which  to  formulate  guidance.  One  such  topic  is  the  integration  of  advanced  HSl  technology 
into  conventional  NPPs.  The  NRC  is  currently  sponsoring  research  at  Brookhaven  National  Laboratory  (BNL)  to 

(1)  better  define  the  effects  of  changes  in  HSls  brought  about  by  incorporating  digital  technology  on  personnel 
performance  and  plant  safety,  and  (2)  develop  HFE  guidance  to  support  safety  reviews,  should  a  review  of  plant 
modifications  or  HSIs  be  necessary.  This  guidance  will  be  integrated  into  NUREG-0700  and  provide  the  NRC’s 
staff  with  the  technical  basis  to  help  ensure  that  HSl  designs  or  plant  modifications  do  not  compromise  safety. 

The  results  of  this  project  are  expected  to  contribute  to  satisfying  the  NRC’s  goals  of  (1)  maintaining  safety, 

(2)  increasing  public  confidence,  (3)  increasing  regulatory  efficiency  and  effectiveness,  and  (4)  reducing 
unnecessary  burden. 

Based  upon  the  literature,  interviews,  and  site  visits,  O'Hara  et  al.  (1996)  identified  changes  in  HSl  technology  and 
their  potential  effects  on  personnel  performance.  The  topics  were  then  evaluated  for  their  potential  safety 
significance  (Stubler  et  al.,  1996);  computer-based  procedures  (CBPs)  was  one  HSl  technology  that  was  found  to  be 
potentially  safety  significant.  (The  safety  analysis  is  described  in  more  detail  in  Section  5.4.2.2  of  this  report.) 

Plant  procedures  provide  instructions  to  guide  operators  in  monitoring,  decision  making,  and  controlling  the  plant. 
Historically,  plant  procedures  have  been  paper-based  and  were  not  considered  part  of  the  HSl.  Following  the 
accident  at  the  Three  Mile  Island  NPP,  the  nuclear  power  industry  recognized  the  importance  of  having 
technologically  sound  and  easy-to-use  procedures  to  handle  major  plant  disturbances.  For  emergency  operations, 
symptom-based  procedures  were  established  that  enabled  operating  crews  to  restore  and  maintain  the  plant’s  safety 
functions  without  having  to  diagnose  events  or  the  specific  causes  of  process  disturbances.  The  NRC  and  industry 
put  a  great  deal  of  effort  into  the  design  and  review  of  emergency  operating  procedures  (EOPs)  (American  Nuclear 
Society,  1981;  Barnes  et  al.,  1989;  Galletti  and  Sutthoff,  1992;  NRC,  1982).  More  recently,  studies  of  other 
operating  procedures  (e.g.,  normal  and  abnormal  procedures)  also  have  demonstrated  the  importance  of  these 
categories  of  procedures  to  plant  safety  (Grant  et  al.,  1989). 

Paper-based  procedures  (PBPs)  have  characteristics  that  limit  how  information  can  be  presented  to  the  operators. 
These  limitations  include  presenting  information  in  sequential  form,  requiring  numerous  iterations  through  steps, 
and  cautions  or  warnings  that  may  not  be  applicable  for  all  system  states  (Wourms  and  Rankin,  1994;  Mampaey  et 
al.,  1988).  PBPs  also  impose  tasks  on  the  operator  that  are  not  directly  related  to  controlling  the  plant.  To  make 
transitions  between  procedure  steps  and  documents,  and  maintain  awareness  of  the  status  of  procedures  that  are  in 
progress,  operators  must  handle,  arrange,  scan,  and  read  PBPs  in  parallel  with  monitoring  and  control  tasks. 
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CBPs  are  being  developed  to  support  procedure  management.  CBPs  have  a  range  of  capabilities  that  may  support 
operators  in  controlling  the  plant  and  reduce  the  demands  associated  with  PBPs.  In  their  simplest  form,  CBPs 
show  the  same  information  via  computer-driven  video  display  units  (VDUs).  More  advanced  CBPs  may  include 
features  to  support  managing  procedures  (e.g.,  making  transitions  between  steps  and  documents,  and  maintaining 
awareness  of  procedures  in  progress),  detecting  and  monitoring  the  plant’s  state  and  parameters,  interpreting  its 
status,  and  selecting  actions  and  executing  them. 

CBPs  are  being  developed  for  new  plants,  e.g.,  the  Westinghouse  AP600,  and  as  upgrades  for  existing  plants,  e.g., 
the  Beznau  plant  in  Switzerland.  Although  CBP  systems  are  being  developed,  the  general  consensus  is  that 
guidance  for  their  design  is  lacking  (Chignell  and  Zuberec,  1993;  Converse,  1992;  EPRl,  1993a). 

The  introduction  of  advanced  HSl  technology,  such  as  CBP  systems,  is  generally  considered  to  enhance 
performance,  but  there  also  is  the  potential  to  lower  human  performance,  spawn  new  types  of  human  errors,  and 
reduce  human  reliability  (O'Hara,  1994).  Therefore,  it  is  important  to  consider  the  potential  effects  of  these 
technologies  on  human  performance.  Like  other  advanced  HSI  technologies,  CBPs  have  many  characteristics  that 
can  enhance  a  crew’s  performance,  but  other  characteristics  may  impair  their  responses.  In  addition,  CBP  failures 
may  place  special  demands  on  operators,  e.g.,  transitions  between  CBPs  and  PBPs  may  introduce  problems 
associated  with  their  different  presentation  media  and  requirements  for  operation. 

1.2  Earlier  NRC  Work  on  Computer-Based  Procedures 

As  part  of  their  review  of  advanced  reactors,  the  NRC’s  staff  evaluated  the  Electric  Power  Research  Institute's 
(EPRl)  Advanced  Light  Water  Reactor  Utility  Requirements  Document,  hereafter  referred  to  as  the  URD  (see 
EPRI,  1993a).  EPRl  specified  CBPs  as  a  requirement  in  the  URD,  but  gave  limited  guidance  for  their  development 
and  implementation.  The  NRC  (1994)  concluded  that  CBPs  were  a  “desirable  goal”  whose  appropriate 
implementation  must  be  demonstrated. 

In  1994,  the  NRC  staff  in  the  Office  of  Nuclear  Reactor  Regulation  published  the  Human  Factors  Engineering 
Program  Review  Model  (NUREG-071 1),  giving  an  approach  to  reviewing  the  HFE  aspects  of  advanced  reactor 
designs  (O'Hara,  Higgins,  Stubler,  Goodman,  Eckenrode,  Bongarra,  and  Galletti,  1994).  Criterion  7  of  Element  8, 
Procedure  Development,  of  the  NUREG-071 1  states  the  following: 

An  analysis  should  be  conducted  to  determine  the  impact  of  providing  computer-based  procedures,  CBPs,  (either 
partial  or  complete),  and  to  specify  where  such  an  approach  would  improve  procedure  utilization  and  reduce 
operating  crew  errors  related  to  procedure  use.  The  justification  for  use  of  CBPs  over  paper  procedures  should  be 
documented.  An  analysis  of  alternatives  in  the  event  of  loss  of  CBPs  should  be  performed  and  documented. 

In  supporting  NUREG-071 1,  preliminary  review  guidance  was  developed  for  CBPs  based  upon  the  considering 
current  issues  and  practices  in  using  PBPs  (Barnes  et  al.,  1996). 

From  a  research  perspective,  CBPs  were  identified  in  1994  as  a  technology  being  developed  in  the  nuclear  power 
industry  for  which  little  human  factors  knowledge  and  guidance  existed  (O’Hara,  1 994).  After  this  identification 
of  the  CBPs  issue,  the  literature  was  reviewed  for  the  NRC  (Wourms  and  Rankin,  1994)  and  a  workshop  for 
subject-matter  experts  (SMEs)  was  held  in  San  Diego,  California,  to  identify  the  state-of-the-art  in  CBP  research 
and  design.  The  workshop  generated  an  excellent  overview  of  the  systems  under  development  and  some  of  the 
human  factors  challenges  to  their  use  and  evaluation  (see  Section  5.4.3,  Expert  Opinion,  of  this  report).  Also  in 
1994,  the  NRC  started  a  study  of  CBP  systems  and  their  potential  impact  on  human  and  plant  reliability  (Orvis  and 
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Spurgin,  1996).  It  also  sponsored  a  comparison  of  operators’  performance  with  CBPs  and  PBPs  in  a  simulator 
study  (Converse,  1995). 

As  noted  above,  the  potential  human  performance  issues  and  safety  significance  of  CBPs  recently  were  evaluated 
the  early  phases  of  this  project  (O’Hara,  Stubler,  and  Higgins,  1996;  Stubler,  Higgins,  and  O’Hara,  1996).  Also, 
observations  were  made  on  introducing  advanced  HSI  technology,  including  CBPs,  into  a  conventional  NPP  (Roth 
and  O’Hara,  1998). 

This  work  is  discussed  in  greater  detail  in  Section  5  of  this  report;  all  of  it  contributed  to  the  CBP  review  guidance 
developed. 

1.3  Organization  of  the  Report 

The  report  is  divided  into  two  parts.  Part  1  describes  the  methodology  for  developing  guidance  and  its  technical 
basis.  The  objective  of  the  study  is  described  in  Section  2,  and  the  guidance  development  methodology  in 
Section  3.  Section  4  characterizes  CBP  systems,  and  Section  5  discusses  the  literature  and  information  that  served 
as  the  technical  basis  for  the  review  guidance.  The  actual  way  we  used  the  technical  information  is  described  in 
Section  6.  Our  CBP  research  is  summarized  in  Section  7.  Section  8  lists  the  references  to  the  published  literature. 

Part  2  of  the  document  contains  the  results  of  the  guidance  development,  presented  in  two  sections.  Section  9 
identifies  the  design-process  considerations  for  CBP  review,  and  Section  10  contains  the  HFE  design  guidelines  for 
reviewing  an  implemented  CBP  design. 
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The  objective  of  this  study  was  to  develop  HFE  review  guidance  for  CBP  systems  based  on  a  technically  valid 
methodology.  While  the  primary  focus  of  the  guidance  was  on  EOPs,  many  of  the  principles  identified  apply  to 
other  types  of  plant  procedures. 

To  support  this  objective,  several  tasks  were  performed: 

•  Development  of  a  framework  for  characterizing  key  design  features  of  CBP  systems 

•  Development  of  a  technical  basis  using  research  and  analyses  on  human  performance  relevant  to  CBPs 

•  Development  of  HFE  review  guidelines  for  CBPs  in  a  format  that  is  consistent  with  NUREG-0700,  Rev.  1,  and 
NUREG-0711 

•  Identification  of  remaining  CBP  issues  for  which  research  was  insufficient  to  support  the  development  of  NRC 
review  guidance 
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3.1  Overview 


Figure  3.1  shows  the  overall  methodology  used  for  developing  NUREG-0700  guidance.  The  process  is  discussed 
in  detail  elsewhere  (O’Hara,  Brown,  and  Nasta,  1 996;  Stubler  and  O’Hara,  1 996).  The  portion  of  the  methodology 
applicable  to  this  report  and  project  is  boxed  in  the  figure.  This  section  of  the  report  describes  the  general  rationale 
behind  guidance  development. 


Figure  3.1  Major  Steps  in  the  Development  of  NUREG-0700  Guidance 


The  methodology  was  guided  by  the  following  objectives: 

•  Establish  a  process  that  will  result  in  valid,  technically  defensible,  review  criteria 

•  Establish  a  generalizable  process  applicable  to  any  aspect  of  HSl  technology  needing  review  guidance 

•  Establish  a  process  that  optimally  uses  available  resources;  i.e.,  develop  a  cost-effective  methodology 

The  methodology  places  a  high  priority  on  establishing  the  validity  of  the  guidelines.  Validity  is  defined  along  two 
dimensions:  internal  and  external.  Internal  validity  is  the  degree  to  which  the  individual  guidelines  are  based  on 
an  auditable  technical  basis.  The  technical  basis  is  the  information  upon  which  the  guideline  is  established  and 
justified,  and  varies  for  individual  guidelines.  Some  guidelines  may  be  based  on  technical  conclusions  from  a 
preponderance  of  empirical  research  evidence,  some  on  a  consensus  of  existing  standards,  while  others  are  based 
on  judgement  that  a  guideline  represents  good  practices  based  on  the  information  reviewed.  Maintaining  an  audit 
trail  from  each  guideline  to  its  technical  basis  serves  several  purposes: 

•  Evaluation  of  the  technical  merit  of  the  guideline  by  others 

•  A  more  informed  application  of  the  guideline  since  its  basis  is  available  to  users 

•  Evaluation  of  deviations  or  exceptions  to  the  guideline 

External  validity  is  the  degree  to  which  the  guidelines  are  subjected  to  independent  peer  review.  Peer  review  is  a 
good  method  for  screening  guidelines  for  conformance  to  accepted  HFE  practices,  and  for  comparing  guidelines  to 
the  practical  operational  experience  of  HSl  s  in  real  systems. 
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For  individual  guidelines,  these  forms  of  validity  can  be  inherited  from  the  source  documents  that  form  their 
technical  basis.  Some  HFE  standards  and  guidance  documents,  for  example,  already  have  good  internal  and 
external  validity.  However,  if  validity  is  not  inherited,  it  is  established  as  part  of  the  process  of  guidance 
development.  The  NUREG-0700  methodology  was  established  to  ensure  validity,  both  inherited  from  its  technical 
basis,  and  through  the  development  and  evaluation  of  guidance. 

Figure  3.2  depicts  the  process  used  to  develop  the  technical  basis  and  guidance;  it  emphasizes  information  sources 
with  the  highest  degree  of  internal  and  external  validity.  Thus,  primary  and  secondary  source  documents  were 
sought  for  guidance  first,  followed  by  tertiary  source  documents,  basic  literature,  and  industry  experience,  and  from 
them  design  principles  and  lessons  from  industry  experience  were  identified.  Using  this  technical  basis  as  a 
foundation,  the  guidance  was  developed.  For  specific  aspects  of  the  topic  in  which  there  was  an  inadequate 
technical  basis  to  develop  guidance,  unresolved  research  issues  were  defined.  Thus,  the  analysis  of  information  led 
to  the  formulation  of  both  guidance  and  issues.  The  resulting  guidance  documentation  includes  HFE  guidelines, 
technical  basis,  the  development  methodology,  and  unresolved  research  issues. 

Each  step  in  this  research  -  characterizing  the  topics,  developing  the  technical  basis,  developing  and  documenting 
guidance,  identifying  issues,  and  peer  review  -  is  discussed  in  greater  detail  in  the  sections  that  follow. 

3.2  Characterization  of  CBP  Systems 

The  first  step  in  the  development  process  was  to  identify  the  areas  for  which  guidance  was  needed.  Existing  CBP 
systems  were  reviewed  to  identify  the  features  and  functions  along  which  CBP  systems  can  be  defined. 
Characterization  was  important  because  it  provided  a  structure  with  which  to  organize  the  design  review  guidance. 
The  characterization  will  also  provide  a  reviewer  with  a  framework  for  requesting  information  about  a  CBP 
system.  Section  4  describes  the  characterization  of  CBP  systems. 

3.3  Development  of  Technical  Basis 

The  development  of  detailed  review  guidelines  began  by  collecting  technical  information  on  which  they  would  be 
based  (see  Figure  3.2);  the  process  was  designed  to  develop  valid  guidance  cost  effectively.  First,  primary  source 
documents  were  sought.  These  were  HFE  standards  and  guidance  documents  with  internal  and  external  validity; 
that  is,  these  documents  generally  had  their  own  research  bases,  and  the  developers  of  these  documents  had 
considered  the  available  research  and  operational  experience,  along  with  their  own  expertise,  to  establish  HFE 
guidelines.  These  primary  source  documents  were  extensively  peer  reviewed.  They  were  developed  by  experts  who 
consider  research  in  terms  of  its  applicability  and  general izability  to  real  systems,  include  knowledge  and  expertise 
gained  through  operational  experience  and  the  application  of  guidance,  and  modify  the  guidance  based  on 
extensive  peer  review.  Such  documents  provided  a  technically  valuable  starting  place. 

Since  little  primary  source  information  was  available,  the  technical  basis  for  CBPs  considered  the  other  sources 
identified  in  Figure  3.2.  Secondary  sources  were  documents  for  which  either  internal  or  external  validity  had  been 
established.  They  were  preferred  over  tertiary  source  documents  for  which  neither  was  established. 

In  addition  to  these  sources,  the  results  from  basic  literature  were  analyzed  (articles  from  technical  journals,  reports 
from  research  organizations,  and  papers  from  technical  conferences).  When  guidance  was  based  on  basic 
literature,  engineering  judgement  was  required  to  generalize  from  the  individual  experiments  to  actual  applications 
in  the  workplace  because  individual  experiments  had  unique  constraints  limiting  their  generality  (such  as  their 
unique  participants,  types  of  tasks  performed,  and  types  of  equipment  used).  For  example,  most  scientific 
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Figure  3.2  Technical  Basis  and  Guidance  Development  Process 


experiments  do  not  involve  tasks  of  the  complexity  of  NPP  operations,  nor  do  they  examine  tasks  under  the  same 
performance  shaping  factors  (such  as  rotating  shifts,  stress,  and  fatigue)  as  exist  in  a  work  environment.  While 
information  from  research  is  a  valuable  part  of  guidance  development,  it  usually  cannot  be  uncritically  adopted. 
Thus,  the  results  must  be  interpreted  in  the  context  of  real-world  tasks  and  systems,  based  on  professional  and 
operational  experience. 
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Industry  experience  also  was  used,  such  as  published  case  studies,  surveys,  and  interviews  with  knowledgeable 
experts.  Although  such  information  may  lack  a  rigorous  experimental  basis  (and  thus,  a  measure  of  validity),  it  is 
highly  relevant. 

Finally,  some  issues  were  evaluated  by  original  research.  This  approach  has  the  advantage  of  being  focused  on 
specific  issues  of  interest,  and  has  both  high  relevance  and  a  sound  basis  from  which  to  establish  validity.  The 
study  (Roth  and  O’Hara,  1998)  is  described  in  Section  5.4.1 ,  Empirical  Evaluation  of  CBPs  Based  on  Personnel 
Performance. 


3.4  Development  and  Documentation  of  Guidance 

Once  the  technical  information  was  assembled,  a  draft  set  of  guidelines  was  developed  from  it.  The  guidelines 
were  organized  and  specified  in  a  standard  format  (discussed  in  Section  6).  They  are  identified  in  Part  2  of  this 
document. 

3.5  Identification  of  Issues 

Where  there  was  insufficient  information  to  provide  a  technical  basis  upon  which  to  develop  valid  design  review 
guidance,  an  issue  was  defined;  these  issues  are  described  in  Section  7. 

From  a  research  standpoint,  issues  reflect  aspects  of  CBP  design  and  use  that  require  additional  investigation  to 
resolve.  From  a  design  review  standpoint,  these  issues  will  have  to  be  addressed  case-by-case.  For  example,  an 
issue  can  be  dealt  with  as  part  of  design-specific  tests  and  evaluations. 

3.6  Peer  Review 

The  resulting  technical  basis  and  guidance  was  submitted  for  review  by  knowledgeable  experts.  These  included 
reviews  by  personnel  from  the  U.S.  NRC  with  expertise  in  human  factors  engineering  and  engineering  fields 
directly  related  to  the  topic.  Additional  reviews  were  conducted  by  human  factors  specialists  outside  the  NRC  who 
have  expertise  in  human  performance  in  complex  systems,  such  as  nuclear  power  plants  and  aviation.  These 
external  reviews  included  evaluations  of  the  topic  characterization  along  the  following  criteria:  clarity,  accuracy, 
and  completeness;  and  of  the  review  guidance  along  the  following  criteria:  organization,  necessity,  sufficiency, 
resolution,  and  technical  basis.  Comments  from  the  peer  reviews  were  incorporated  into  the  present  version  of  this 
document. 
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In  the  nuclear  power  industry,  a  procedure  has  been  generally  defined  as  a  written  document  (including  both  text 
and  graphics)  that  presents  a  series  of  decision  and  action  steps  to  be  performed  by  plant  personnel  (e.g.,  operators, 
technicians)  to  accomplish  a  goal  safely  and  efficiently.  NPP  personnel  use  procedures  for  a  wide  variety  of  tasks, 
from  administration  to  testing,  and  plant  operation.  This  project  is  focused  on  procedures  that  prescribe 
interactions  between  personnel  and  the  plant  systems  and  components.  The  purpose  of  NPP  procedures  is  to  guide 
human  actions  when  performing  a  task  to  increase  the  likelihood  that  the  actions  will  safely  achieve  the  task’s 
goal.  In  contrast  to  decision  aids,  procedures  define  decisions  to  be  made  and  actions  to  be  taken  where  the  task 
goals  are  unambiguous  and  the  correct  or  desired  course  of  action  is  generally  known. 

In  recent  years,  many  efforts  have  been  started  in  NPPs  to  assist  personnel  through  the  computerization  of 
procedure  information.  Several  CBP  systems  have  been,  or  are  being,  installed  in  operating  plants  or  in  their 
training  simulators.  The  following  are  examples  of  some  of  the  more  mature  systems: 

•  Westinghouse  Computerized  Procedure  (COMPRO)  System  at  Beznau,  Switzerland,  and  Temelin,  Czech 
Republic 

•  EdF  Computerized  Control  Room  (CR)  for  N4  Reactors  at  Chooze  and  Civaux,  France 

•  EPRI  Boiling  Water  Reactor  (BWR)  Emergency  Operating  Procedure  Tracking  System  (EOPTS)  at  Kuosheng, 
Taiwan 

•  Tokyo  Electric’s  BWR  Computerized  EOPs,  France 

(For  a  general  description  of  specific  CBP  systems,  see  Moieni  and  Spurgin,  1993a;  Spurgin,  Wachtel,  and  Moieni, 
1993.) 

For  this  document,  CBPs  are  defined  narrowly  to  include  computer  systems  whose  purpose  is  supporting  the 
presentation  and  use  of  procedures;  systems  whose  functions  include  diagnosis  or  disturbance  analysis  are  not 
within  its  scope.  The  focus  of  the  effort  is  on  the  HFE  aspects  of  CBPs,  not  the  software  aspects  (for  a  discussion  of 
general  software  development,  testing,  and  management,  see  NRC  Regulatory  Guides  1.168  through  1.173;  NRC, 
1997  a-f). 

The  characterization  and  guidance  focuses  on  EOPs.  However,  it  is  recognized  that  normal  and  abnormal 
operating  procedures  have  been  identified  as  important  contributors  to  many  significant  events  (Trager,  1988),  and 
play  a  significant  role  in  the  plant’s  safety  (Grant  et  al.,  1989).  Much  of  the  guidance  may  apply  to  such 
procedures,  and  to  test,  surveillance,  troubleshooting,  and  maintenance  procedures  when  they  are  delivered  by  CBP 
systems. 

The  design  review  of  CBP  systems  requires  two  types  of  guidance:  procedure  guidance  and  HSI  guidance.  The 
first  addresses  the  human  factors  aspects  of  procedure  design  and  is  intended  to  ensure  that  technically  correct  and 
usable  procedures  are  developed.  There  is  considerable  guidance  on  procedure  design,  such  as  NUREG-0899,  but 
because  it  was  developed  for  PBPs,  modifications  may  be  necessitated  by  computerization  of  CBPs.  Sections  9  and 
10  of  this  report  have  guidance  specific  to  CBPs. 

For  HSI  guidance,  CBPs  will  share  many  of  the  HSI  resources  and  characteristics  as  other  plant  information 
systems.  That  is,  information  will  be  presented  on  VDUs,  and  operators  will  interact  with  the  CBP  information 
using  the  computer’s  dialogue  and  navigation  facilities,  accessed  with  input  devices,  such  as  keyboards  and  mice. 
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Many  human  factors  guidelines  currently  exist,  such  as  NUREG-0700,  Rev.  1,  covering  these  general 
characteristics  of  HSI  design.  What  still  is  needed  is  the  specific  application  of  HSI  principles  to  the 
computerization  of  procedure  functions,  such  as  monitoring  steps  that  are  continuously  applicable. 

In  this  section,  a  characterization  framework  for  CBP  systems  is  discussed  based  on  our  examination  of  many 
CBP-system  implementations.  A  system  characterization  is  the  identification  of  important  design  features  and 
functions  that  can  be  used  to  describe  it.  The  cheiracterization  provides  a  framework  for  NRC’s  reviewers  to  collect 
information  about  the  system  for  reviewing  its  design.  It  also  forms  an  organizational  structure  for  the  guidelines 
used  to  review  the  system. 

The  CBP  characterization  framework  discussed  includes  the  following: 

•  Representation  of  Procedures  (Section  4.1) 

•  Functionality  of  Procedures  (Section  4.2) 

•  Management  and  Support  of  Procedures  (Section  4.3) 

•  CBP  Hardware  (Section  4.4) 

•  Backup  System  for  Procedures  (Section  4.5) 

•  Integration  with  Other  HSI  Components  (Section  4.6) 

4.1  Representation  of  Procedures 

In  their  basic  form,  procedures  have  a  number  of  elements  for  which  considerable  guidance  already  exists.  These 
same  elements  must  be  represented  in  the  CBP  system. 

Identification  Information  for  Procedures 


Procedures  are  identifiable  to  the  operators  and  maintainers  through  their  title,  procedure  number,  revision 
number,  and  date.  Procedures  also  contain  statements  of  the  high-level  objective  and  its  applicability,  including 
their  category,  e.g.,  emergency  or  abnormal. 

Basic  Steps 

Steps  are  the  basic  unit  of  the  procedure.  A  basic  action  step  is  composed  of  a  verb  and  a  direct  object.  In  general, 
the  rules  of  English  grammar  are  followed  and  the  syntax  reflects  concise  language  that  is  simply  stated,  explicit, 
and  consistent.  Decision  steps  give  instructions  for  evaluating  conditions  and  for  then  choosing  the  appropriate 
action(s)  from  a  predefined  set.  The  decisions  may  involve  conditional  logic,  i.e.,  where  the  actions  are  to  be 
performed  only  if  a  specified  set  of  conditions  exists.  Action  steps  identify  actions  to  be  taken,  i.e.,  instructions  to 
perform  physical  steps  (e.g.,  “Depress”)  and  mental  ones  (e.g.,  “Verify”);  they  also  describe  the  objectives  of  those 
actions.  Some  procedure  steps  (e.g.,  in  EOPs)  have  a  dual  nature,  with  an  action  to  be  accomplished  in  one 
column,  and  a  second  action  if  the  first  is  not  successful.  Some  steps  may  require  calculations. 

Implementation  of  procedures  has  a  temporal  flow,  i.e.,  some  steps  are  carried  out  when  encountered,  others  are 
continuous  (steps  of  continuous  applicability),  while  time  or  process  criteria  determine  when  others  are  undertaken. 
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Performance  of  a  procedure  step  may  be  supported  by  information,  such  as  cautions  and  notes,  that  qualifies  the 
required  actions  and  decisions. 

Warnings,  Cautions.  Notes,  and  Supplementary  Information 

Warnings  alert  operators  to  potential  hazards  of  their  actions  that  may  cause  death  or  injury  to  workers  or  the 
public;  cautions  alert  operators  to  potential  hazards  for  machinery  or  equipment.  Notes  call  attention  to  important 
supplemental  information  that  may  enhance  an  operator's  understanding  and  performance  of  the  procedure. 

Procedure  steps  may  cite  supporting  supplementary  material  helping  the  op>erator  implement  the  step;  this  material 
may  be  tables,  figures,  lists,  text,  or  numeric  information. 

Lists 


A  list  is  a  display  containing  alphanumeric  strings  arranged  in  a  single  column.  Procedures  frequently  use  list 
formats  to  present  groups  of  items,  such  as  actions,  conditions,  comp>onents,  criteria,  and  systems.  When  lists  are 
used  in  CBPs,  additional  considerations  relate  to  the  grouping  of  items,  provision  of  checkoff  capability,  and 
operator  alerts  to  potentially  overlooked  items. 

Organization  of  Procedures 


NPP  procedures  are  not  simple  checklists  where  the  operator  starts  at  the  top  and  linearly  proceeds  step-by-step  to 
the  end.  Based  on  the  plant’s  conditions,  the  operator  may  be  required  to  branch  from  one  part  of  a  procedure  to 
another,  or  from  one  procedure  to  another.  Thus,  the  way  procedures  are  organized  is  important. 

Format  and  Screen  Layout 

PBPs  generally  present  the  basic  steps  in  text  or  flowchart  formats.  The  CBP  systems  that  have  been  designed  also 
follow  these  principles,  and  may  use  either  format.  Thus,  the  Westinghouse  COMPRO  CBP  is  text  based  and  is 
consistent  with  the  two-column  format  developed  by  the  Westinghouse  Owner’s  Group  for  their  EOPs.  The  EdF 
N4  CBP  uses  a  flowchart  format. 

Unlike  PBPs,  CBPs  are  viewed  through  the  limited  display  area  of  one  or  more  VDUs.  Thus,  whether  the  format  is 
text  or  flowchart,  the  designer  must  decide  whether  the  procedure  will  be  presented  in  a  continuous,  scrollable 
display  or  be  divided  into  discrete  pages. 

The  overall  layout  of  the  screen  for  showing  elements  of  the  procedure  refers  to  the  determination  on  what 
information  should  be  continuously  presented,  and  the  manner  in  which  individual  elements  are  presented. 

For  example,  the  procedure’s  title  and  identification  information  may  be  displayed  continuously  at  the  top  of  the 
CBP  screen,  while  the  steps  are  shown  on  a  scrollable  window.  Cautions  may  be  represented  in  a  separate  window. 
Supporting  features,  such  as  bookmarks,  checklists,  and  operators’  comments  may  also  be  displayed. 

Presentation  formats,  such  as  text  and  flowcharts,  can  be  enhanced  by  the  coding  capabilities  of  computer-based 
displays,  e.g.,  color,  flashing,  animation,  and  auditory  cuing,  which  enhance  the  salience  of  important  information. 
CBPs  use  coding  for  conditions  such  as: 

•  Whether  procedure  step  logic  is  satisfied  or  not 
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•  Whether  information  is  static  or  dynamic  with  the  plant’s  state 

•  When  a  caution  is  in  effect 

•  When  a  change  occurs  in  the  status  of  a  continuously  monitored  step 

CBPs  can  be  designed  to  allow  operators  to  choose  the  level  of  detail  shown.  For  example,  operators  may  choose  to 
have  less  detail  presented  when  a  procedure  step  is  satisfied.  Alternatively,  an  operator  may  choose  to  see  all  of  the 
individual  evaluations  that  led  to  the  conclusion  that  the  step  is  satisfied. 

4.2  Functionality  of  Procedures 

A  significant  difference  between  PBPs  and  CBPs  is  in  the  functionality  provided  by  the  latter.  Procedure  functions 
can  be  organized  into  four  cognitive  categories:  monitoring  and  detection,  situation  assessment,  response  planning, 
and  response  implementation  (see  Section  5.1  for  an  in-depth  discussion  of  these  cognitive  functions).  In 
monitoring  and  detection,  operators  must  monitor  the  process  parameters  referenced  by  procedures,  and  also  their 
own  actions  in  response. 

Situation  assessment  is  frequently  required  by  procedures.  While  EOPs  enable  operators  to  act  without  diagnosing 
the  disturbance,  operators  must  assess  whether  EOP  entry  conditions  exist.  Within  the  procedure,  operators  assess 
each  decision  step  by  comparing  actual  values  to  the  reference  values,  evaluating  whether  cautions  are  applicable, 
assessing  the  completeness  of  each  step,  and  tracking  and  remembering  their  path  through  the  procedure  (the 
procedure  history);  at  the  same  time,  they  must  evaluate  steps  of  continuous  applicability  and  steps  that  are  time  or 
parameter  dependent.  Operators  also  must  assess  the  applicability  of  individual  steps  because  PBPs  are  generic 
and  not  context  sensitive  (context  sensitivity  is  the  selection  of  procedural  information  based  on  the  plant’s  state). 
Finally,  operators  must  evaluate  the  success  of  the  current  procedure  in  achieving  the  high-level  procedure  goals, 
and  the  procedure’s  termination  conditions. 

Procedures  were  originally  designed  to  support  response  planning.  For  example,  EOPs  assist  operators  in 
responding  to  events  by  setting  out  the  steps  necessary  to  achieve  safety  goals.  They  relieve  the  operator  of  the 
burden  of  formulating  response  plans  in  real  time.  Instead,  the  actions  necessary  to  restore  and  maintain  critical 
safety  functions  were  analyzed  in  advance  and  developed  into  a  set  of  detailed  procedures.  However,  operators 
must  still  evaluate  whether  transitions  to  other  parts  of  the  procedure  or  other  procedures  are  warranted.  Rarely, 
they  may  have  to  modify  procedures  when  current  conditions  render  the  existing  procedure  inapplicable  (see 
Section  5.1). 

With  respect  to  response  implementation,  the  operator’s  responses  involve  acting  upon  the  procedures  themselves, 
such  as  making  the  transition  from  one  step  to  the  next,  to  other  parts  of  the  procedure,  or  to  other  procedures. 
Responses  also  include  controlling  equipment  based  on  procedural  guidance. 

While  PBPs  support  response  planning,  they  give  little  active  support  for  monitoring,  situation  assessment,  and 
responses.  On  the  other  hand,  CBPs  may  support  these  cognitive  functions  as  well;  the  extent  to  which  they  do  so 
depends  on  their  design.  Examining  the  role  of  the  operator  is  very  significant  in  defining  how  cognitive  functions 
are  supported  by  CBP  design  (see  Section  5). 

Table  4.1  provides  an  overall  scheme  within  which  the  level  of  automation  of  CBPs  can  be  organized.  It  illustrates 
the  wide  levels  of  automation  and  functionality  that  CBPs  may  possess.  The  table  also  can  be  used  to  catalog  the 
functionality  of  a  particular  CBP  system. 
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In  the  rows,  the  general  cognitive  functions  (as  described  above)  are  identified,  along  with  the  associated 
procedure-related  activities.  In  the  columns,  four  levels  of  automation  are  identified;  manual,  advisory,  shared, 
and  automated: 

•  Manual  -  The  function  is  performed  by  the  operators  with  no  assistance  from  the  CBP. 

•  Advisory  -  The  CBP  provides  advice  only.  For  example,  it  may  advise  the  operator  that  Pump  A  should  be 
started,  but  does  not  start  it. 

•  Shared  -  The  CBP  and  the  operators  both  perform  the  function.  For  example,  a  CBP  system  may  perform 
process  monitoring  but  may  not  monitor  all  information  about  the  system,  such  as  a  valve’s  position,  because  it 
lacks  the  instrumentation.  When  this  type  of  information  needs  to  be  monitored,  the  operator  provides  it. 

Table  4.1  Levels  of  Automation  of  Procedure  Functions 


Procedure  Functions 

Level  of  Automation 

Manual  Advisory  Shared  Automatic 

Monitoring  and  Detection 

Process  parameter  values 

NA 

Op)erator  actions 

NA 

Situation  Assessment 

Procedure  entry  conditions 

NA 

Resolution  of  procedure  step  logic 

NA 

Step  status  (incomplete  or  completed) 

NA 

Procedure  history 

NA 

Context  sensitive  step  presentation 

NA 

Assessment  of  continuous,  time,  and  parameter  steps 

NA 

Assessment  of  cautions 

NA 

High-level  goal  attainment  and  procedure  exit 
conditions 

NA 

Response  Planning 

Selection  of  next  step  or  procedure 

Procedure  modification  based  on  current  situation 

i  Response  Implementation 

Transition  from  one  step  to  the  next 

Transition  to  other  parts  of  procedure  or  to  other 
procedures 

Control  of  plant  equipment 

Note:  NA  means  “not  applicable.”  For  a  given  CBP  system,  the  advisory  level  of  automation  may  not  be  applicable  or 

an  entire  function  may  not  be  applicable. 
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•  Automated  -  The  CBP  performs  the  function  automatically  without  the  operator’s  direct  intervention;  the 
operators  may  or  may  not  be  notified  of  the  actions  taken. 

A  given  level  of  automation  does  not  necessarily  apply  to  all  functions.  For  example,  for  process  monitoring,  it  is 
not  meaningful  to  have  advisory  automation.  The  CBP  system  will  either  have  monitoring  capability  or  not. 

These  are  indicated  by  NA  (not  applicable)  in  the  table. 

A  given  procedure  system  may  not  provide  an  entire  function.  For  example,  a  particular  CBP  may  not  address  the 
control  of  equipment  in  any  capacity,  not  even  manual;  equipment  would  be  operated  from  other  HSIs. 

Individual  CBP  systems  differ  in  terms  of  their  levels  of  automation  (i.e.,  the  extent  to  which  they  provide  features 
beyond  those  identified  above  as  the  basic  procedure  elements).  For  example,  to  allow  manual  control  of 
components,  the  CBP  must  include  a  control,  e.g,,  a  soft  control,  for  that  equipment. 

4.3  Management  and  Support  of  Procedures 

CBP  systems  have  design  features  that  support  operators’  interaction  with  the  system,  procedure  maintenance,  and 
configuration  control.  Therefore,  interface  management  features  (such  navigation  aids)*  are  part  of  the 
characterization  of  CBP  systems.  Procedure-specific  management  support  includes  HSIs  to  transition  between 
procedure  steps  and  between  different  procedures.  The  use  of  procedures  can  be  supported  by  facilities  to  monitor 
and  record  the  operator’s  actions  and  to  provide  help. 

Maintenance  of  procedures  and  configuration  control  are  important  for  CBPs,  as  they  are  for  PBPs,  However,  their 
mechanisms  are  likely  to  differ,  such  as  how  procedures  are  entered  into  the  computer  system,  how  their  quality  is 
verified  (e.g.,  no  typos  or  omissions),  how  errors  in  the  CBPs  are  identified,  tracked  and  corrected,  how  changes 
are  incorporated,  and  how  configuration  is  controlled.  Guidance  on  these  aspects  is  not  part  of  this  project. 

Many  general  interface  management  design  features  are  addressed  in  NUREG-0700.  More  specific  guidance  to 
address  soft  controls  (Stubler,  O’Hara,  and  Kramer,  2000)  and  interface  management  (O'Hara,  Stubler,  and  Nasta, 
1997)  is  being  developed. 

4.4  CBP  Hardware 

CBPs  Utilize  CR  devices  such  as  VDUs,  printers,  and  computer  input  devices,  such  as  alphanumeric  keyboards, 
trackballs,  mice,  and  touch  screens  that  are  part  of  the  CBP  characterization.  NUREG-0700,  Rev.  1  has  guidance 
for  their  review. 

4.5  Backup  System  for  Procedures 

CBPs  can  fail  or  malfunction.  When  important  operations  cannot  be  suspended  or  put  off  while  the  system  is 
repaired,  backup  to  the  CBP  is  needed.  For  EOPs,  a  delay  in  operations  during  a  failure  is  unacceptable;  therefore, 
some  form  of  procedure  backup  is  warranted. 


*  Guidance  for  interface  management  review  is  currently  being  developed  in  a  separate  NRC  research 

project. 
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4.6  Integration  with  Other  HSI  Components 

Integration  of  the  CBP  with  other  CR  HSls  must  be  considered.  Depending  on  the  level  of  automation  (see 
Table  4.1),  CBP  systems  require  varying  types  of  interconnection  with  the  remainder  of  the  CR  HSI.  Their 
consistency  and  compatibility  with  other  HSI  components  can  affect  operators’  performance.  Thus,  important 
considerations  in  reviewing  CBPs  include  the  degree  to  which  (1)  the  display  of  plant  variables  in  the  CBP  is 
compatible  with  normal  monitoring  displays,  (2)  coding  schemes  are  compatible,  and  (3)  control  modes  of  the  CBP 
are  consistent  with  the  rest  of  the  HSI  (e.g.,  with  the  modes  of  automated  control  systems). 
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The  purpose  of  this  section  is  to  establish  a  technical  basis  for  developing  CBP  guidelines  and  to  identify  human 
performance  issues.  The  review  considers  human  performance  research  that  contributes  to  understanding  CBP 
design  and  operational  use.  The  research  included  CBP  use  in  the  NPP  industry  and  several  related  areas:  issues 
associated  with  PBPs,  computerization  of  tasks  traditionally  using  paper  performance  aids,  and  computerized 
operator  support  systems  (COSSs).  In  addition,  we  consider  the  general  cognitive  functions  associated  with  the 
supervisory  control  tasks  which  procedures  support. 


5.1  Cognitive  Tasks  Associated  with  Operating  the  Plant 

Operators  contribute  to  the  plant's  defense-in-depth  approach  to  safety,  serving  a  vital  function  in  ensuring  its  safe 
operation.  However,  they  may  impact  safety  by  making  errors.  Basically,  an  error  occurs  when  personnel  do  not 
perform  a  safety-related  action  within  the  time  required  (sometimes  called  an  error  of  omission).  An  error  also 
may  occur  because  personnel  have  an  incorrect  understanding  of  conditions  and  take  the  wrong  action  (an  error  of 
commission).  Many  attempts  were  made  over  the  past  20  years  to  identify  the  causes  of  error.  The  main 
conclusion  is  that  few  errors  represent  random  events;  instead,  most  can  be  explained  by  human  cognitive 
mechanisms  (Reason,  1988;  Rasmussen,  1986).  Therefore,  it  is  important  to  understand  how  operators  process 
information  and  how  this  relates  to  HSl  design  and  human  error. 

The  operator's  role  in  an  NPP  is  that  of  a  supervisory  controller,  i.e.,  the  plant’s  performance  results  from  the 
interaction  of  human  and  automatic  control.  Reason  (1990)  called  this  a  complex  multiple-dynamic  configuration, 
which  is  difficult  for  personnel  to  handle  when  things  go  wrong.  In  addition  to  process  failures,  automatic  control 
systems  and  the  HSl  also  can  fail.  Thus,  personnel  must  respond  to  plant  failures  and  to  the  interfaces  that 
communicate  their  occurrence.  One  significant  aspect  of  the  HSl  in  responding  to  process  failures  is  the  procedure 
system.  In  complex  systems  using  a  defense-in-depth  philosophy  operations  are  analyzed  in  advance  to  provide 
procedural  support  for  both  normal  and  abnormal  events.  However,  even  when  procedures  are  used,  operators 
must  still  engage  in  higher-level  cognitive  functioning  (Dien,  Montmayeul,  and  Beltranda,  1991;  Roth,  Mumaw, 
and  Lewis,  1994). 

The  operators’  impact  on  the  plant’s  functions,  processes,  systems,  and  components  is  mediated  by  a  causal  chain 
from  their  physiological  and  cognitive  processes,  to  task  performance,  and  ultimately,  to  the  plant’s  performance 
through  the  operators’  manipulation  of  the  HSl  (see  Figure  5.1).  HSl  design,  including  its  procedures,  affects  the 
plant’s  performance  through  personnel  tasks  that  support  operations.  Conceptually,  the  role  of  personnel  can 
involve  two  types  of  tasks.  Primary  tasks  are  those  the  operator  performs  as  part  of  the  functional  role  of 
supervising  the  plant.  Operators  may  be  required  to  act  in  support  of  the  plant's  performance  of  a  higher-level 
function.  Even  when  they  are  not  required  to  take  an  explicit  action,  they  must  monitor  the  performance  of 
automatic  systems  and  intervene  when  the  systems  fail  or  perform  at  unacceptable  levels. 

Primary  tasks  involve  several  generic  cognitive  tasks;  i.e.,  situation  assessment,  monitoring  and  detection,  response 
planning,  and  response  implementation  (see  Figure  5.2).  For  primary  tasks,  these  generic  cognitive  tasks  are 
discussed,  rather  than  the  detailed  specific  tasks,  such  as  monitoring  steam  flow,  starting  pumps,  and  aligning 
valves.  Secondary  tasks  are  those  the  operator  must  perform  when  interacting  with  the  HSl  s  or  job  performance 
aids,  but  which  are  not  directed  to  the  primary  task  (O'Hara,  Stubler,  and  Nasta,  1997).  They  include  navigating 
through  an  information  system  and  manipulating  windows  on  a  VDU.  To  adequately  perform  both  primary  and 
secondary  tasks,  operators  use  their  information  processing  resources,  such  as  attention,  reasoning,  and  memory. 
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Figure  5.1  Hierarchical  Influence  of  Human 
Activity  on  Plant  Performance 


Figure  5.2  Generic  Primary  Tasks  of  a  Supervisory  Controller 
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In  their  traditional  paper  form,  plant  procedures  mainly  support  response  planning.  On  the  other  hand,  CBPs 
potentially  can  affect  all  the  generic  cognitive  primary  and  secondary  tasks  that  operators  undertake: 

•  Monitoring  and  detection,  especially  monitoring  parameters  used  to  evaluate  procedure  steps,  monitoring  steps 
of  continuous  applicability,  and  detecting  violations  of  the  conditions  specified  in  these  steps 

•  Situation  assessment,  especially  assessing  the  plant’s  state  with  respect  to  the  steps’  logic,  steps  of  continuous 
applicability,  and  cautions 

•  Response  planning,  the  main  function  of  procedures 

•  Response  implementation,  for  manual  or  automatic  control  of  (1)  procedure  flow  (transitions  within  and 
between  procedures),  or  (2)  plant  equipment 

Situation  assessment  and  response  planning  are  discussed  next  because  they  are  the  most  important  and 
complicated  cognitive  functions  involved  in  using  procedures.  Monitoring  and  response  implementation  are 
described  afterwards. 

Situation  Assessment 


When  operators  observe  indications  of  an  abnormal  occurrence,  they  try  to  construct  a  coherent,  logical 
explanation  for  them.  This  cognitive  activity  may  be  called  situation  assessment  and  involves  two  related  concepts: 
the  situation  model,  and  the  mental  model.  Operators  develop  and  update  a  mental  representation,  or  so-called 
situation  models  of  factors  known  or  hypothesized  to  be  affecting  the  plant’s  state  at  a  point  in  time.  The  situation 
model  is  the  person’s  understanding  of  the  specific  situation,  and  the  model  is  constantly  updated  as  new 
information  is  received.  To  construct  a  situation  model,  operators  use  their  general  knowledge  and  understanding 
about  the  plant  and  its  operation  to  interpret  information  and  understand  its  implications.  Limitations  in 
knowledge  may  result  in  incomplete  or  inaccurate  situation  models  and  response  plans. 

The  general  knowledge  governing  the  performance  of  highly  exi:>erienced  individuals  may  be  referred  to  as  a 
mental  model  which  constitutes  the  operator's  internal  representation  of  the  physical  and  functional  characteristics 
of  the  system  and  its  operation.  Mental  models  may  not  always  be  accurate  or  complete.  The  mental  model  is  built 
up  through  formal  education,  system-specific  training,  and  operational  experience;  it  resides  in  the  knowledge 
bases  of  long-term  memory.  An  accurate  mental  model  is  a  defining  characteristic  of  expert  performance  (e.g., 
Wickens,  1984;  Bainbridge,  1986;  Moray,  1986;  Rasmussen,  1983;  Sheridan,  1976)  and  is  extremely  important  to 
many  aspects  of  information  processing.  It  is  thought  to  drive  skill-based  processing,  control  rule-based  activity 
through  the  mediation  of  the  operator's  conscious  effort  in  working  memory,  and  provide  the  substantive  capability 
to  reason  and  predict  future  plant  states  required  of  knowledge-based  processing  (Rasmussen,  1983). 

The  distinctions  between  the  mental  and  situation  models  reflect  their  cognitive  basis  in  long-term  and  working 
memory,  respectively.  The  mental  model  is  relatively  permanent.  By  contrast,  an  operator's  situation  model  is  the 
current  interpretation  of  the  plant's  status  and,  therefore,  changeable. 

When  the  operator’s  situation  model  accurately  reflects  the  plant's  state,  the  operator  has  good  situation  awareness. 
Thus,  the  accuracy  of  situation  awareness  depends  on  the  correlation  between  the  operator's  situation  model  and 
the  actual  conditions.  An  operator  can  have  a  good  mental  model  (e.g.,  knowledge  of  how  the  plant  functions)  but 
poor  situation  awareness  because  the  situation  model  does  not  match  the  current  conditions.  Endsley  (1988) 
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identified  situation  assessment  as  the  single  most  important  factor  in  improving  a  crew’s  effectiveness  in  complex 
systems.  Endsley  (1995)  distinguishes  three  levels  of  situation  awareness  (discussed  below). 

An  experienced,  well-trained  operator  easily  develops  an  accurate  situation  model  when  the  HSl  provides 
information  that  readily  maps  to  knowledge  in  the  operator’s  mental  model.  If  no  easy  match  can  be  made  between 
them,  then  situation  assessment  will  require  more  working  memory  and  attention,  and  cognitive  workload  will  be 
high  (Endsley,  1993,  1995;  Fraker,  1988).  However,  in  addition  to  supporting  situation  assessment,  working 
memory  must  support  other  activities,  such  as  selecting  and  taking  actions.  Accordingly,  if  other  tasks  place  high 
demands  on  working  memory,  situation  awareness  may  suffer. 

Situation  awareness  and  cognitive  workload  may  vary  inversely  under  complex,  ambiguous  situations.  For 
example,  under  unfamiliar  or  otherwise  difficult  conditions,  a  high  cognitive  workload  may  entail  decreased 
situation  awareness,  possibly  due  to  a  lack  of  available  attentional  resources  for  analyzing  the  situation.  However, 
Endsley  (1993)  points  out  that  situation  awareness  and  cognitive  workload,  while  interrelated,  may  vary 
independently.  For  example,  a  task  may  be  intensive,  but  readily  recognizable.  Situation  awareness  demands 
cognitive  resources  that  contribute  to  workload,  but  is  not  the  only  cognitive  activity  requiring  such  resources. 

Thus,  mental  models  enable  operators  to  engage  in  situation  assessment  and  to  establish  situation  models.  Good 
situation  models  include  a  knowledge  of  the  important  elements  of  the  current  situation,  and  a  comprehension  of 
how  they  interrelate  to  reflect  the  overall  situation.  These  two  aspects  of  good  situation  models  correspond  to 
Endsley’s  (1995)  Level  1  (Perception  of  Elements)  and  Level  2  (Comprehension  of  Situation)  situation  awareness. 

Mental  models  enable  operators  to  make  predictions  and  form  expectations;  projection  of  future  states  corresponds 
to  Endsley’s  (1995)  Level  3  situation  awareness.  These  expectations  guide  monitoring  and  affect  how  information 
is  interpreted.  This  is  a  general  characteristic  of  information  processing;  it  is  a  synthesis  of  “bottom-up” 
processing  (what  an  operator  perceives  from  the  environment)  and  “top-down”  processing  (what  an  operator 
expects)  (Neisser,  1967).  An  example  of  bottom-up  processing  occurs  during  a  disturbance  when  an  operator 
monitors  the  HSl  and  processes  data  from  the  interface  to  determine  what  is  wrong.  Simultaneously,  these  data  are 
used  to  formulate  hypotheses  or  expectations  about  the  plant’s  status  that  structure  the  perceptual  process  and  data- 
gathering  at  lower  levels.  This  is  top-down  processing.  Both  contribute  to  the  operator's  interpretation  of  the 
situation. 

The  ability  to  predict  from  a  mental  model  based  on  the  current  situation  model  facilitates  “op)en-loop” 
performance  (Moray,  1 986).  “Open-loop”  in  this  context  means  that  behavior  becomes  less  driven  by  feedback  and 
more  governed  by  the  operator’s  prediction  of  future  system  behavior  and  the  desired  goal.  An  NPP  mental  model 
includes  such  knowledge  as  the  physical  interconnections  among  plant  systems  to  predict  flow  paths  (e.g., 
considering  piping  and  valve  interconnections  to  figure  out  how  water  from  one  system  could  get  into  another), 
and  knowledge  of  mass  and  energy  changes  in  one  system  to  predict  the  effect  on  a  second  system  (e.g.,  predicting 
the  effect  that  changes  in  the  secondary  side  steam  generator  levels  and  temperatures  will  have  on  cooldown  of  the 
primary  system).  While  mental  models  provide  the  principles  upon  which  predictions  are  made,  the  situation 
model  provides  the  starting  point  and  is  the  basis  for  developing  expectations  about  events  that  should  be 
happening  at  the  same  time,  how  they  should  evolve  over  time,  and  any  future  effects. 

The  operator’s  expectations  of  the  near-term  future  of  the  plant  guide  the  sampling  of  indicators  to  confirm  the 
inference  (Bainbridge,  1974).  Expectations  are  used  to  search  for  evidence  to  confirm  the  current  situation  model, 
and  to  explain  observed  symptoms.  If  a  new  symptom  is  consistent  with  the  operator’s  expectations,  a  ready 
explanation  for  it  will  be  developed,  yielding  greater  confidence  in  the  situation  model. 
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While  the  mental  model  allows  prediction  and  expectancy  to  guide  control  responses,  expectancy  may  confound  the 
detection  of  subtle  system  failures  (Wickens  and  Kessel,  1981).  When  a  new  symptom  is  inconsistent  with  an 
operator's  expectations,  it  may  be  discounted  or  misinterpreted  in  a  way  to  make  it  consistent  with  the  expectations 
of  the  current  situation  model.  For  example,  an  operator  may  fail  to  detect  key  signals,  or  detect  them  but 
misinterpret  or  discount  them,  because  of  an  inappropriate  understanding  of  the  situation  and  the  derived 
expectations.  Operators  tend  to  ignore  or  discount  symptoms  that  are  not  consistent  with  their  situation  model. 
However,  if  the  new  symptom  is  recognized  as  an  unexpected  behavior,  the  need  to  revise  the  situation  model  may 
become  apparent.  In  that  case,  the  symptom  may  trigger  situation  assessment  activity  to  better  explain  current 
observations.  In  turn,  situation  assessment  may  involve  developing  a  hypothesis  for  what  might  be  occurring,  and 
then  searching  for  confirmatory  evidence.  Thus,  situation  assessment  can  result  in  the  detection  of  abnormal  plant 
behavior  and  of  symptoms  and  alarms  that  otherwise  might  not  have  been  observed  or  might  have  been  missed, 
and  the  identification  of  problems  such  as  sensor  failures  or  plant  malfunctions. 

The  situation  model  is  constantly  updated  as  new  information  is  received  and  a  person’s  understanding  of  a 
situation  changes.  In  NPPs,  maintaining  and  updating  a  situation  model  entails  tracking  the  changing  factors 
influencing  plant  processes,  including  faults,  operators’  actions,  and  automatic  system  responses. 

The  importance  of  mental  and  situation  models,  and  the  expectations  generated,  cannot  be  overemphasized.  They 
not  only  govern  situation  assessment,  but  play  an  important  role  in  guiding  monitoring,  using  procedures  and 
formulating  response  plans,  and  implementing  responses. 

Response  Planning 

Response  planning  refers  to  deciding  upon  a  course  of  action  to  address  an  event.  Response  planning  can  be  as 
simple  as  selecting  an  alarm  response  or  EOP,  or  it  may  involve  thoroughly  developing  a  plan  when  existing 
procedures  have  proved  incomplete  or  ineffective. 

In  general,  response  planning  involves  operators  using  their  situation  model  to  identify  goal  states  and  the 
transformations  required  to  achieve  them.  The  goal  state  may  be  varied,  such  as  to  identify  the  proper  procedure, 
assess  the  status  of  back-up  systems,  or  diagnose  a  problem  (Rasmussen,  1981).  To  achieve  the  goals,  operators 
generate  alternative  response  plans,  evaluate  them,  and  select  the  one  most  appropriate  to  the  current  situation 
model. 

This  is  the  basic  sequence  of  cognitive  activities  in  response  planning;  one  or  more  of  these  steps  may  be  skipped 
or  modified  based  on  the  operator's  assessment  in  a  particular  situation.  When  available  procedures  are  judged 
appropriate  to  the  current  situation,  the  need  to  generate  a  response  plan  in  real-time  may  be  largely  eliminated. 
However,  even  then,  some  aspects  of  response  planning  will  be  undertaken.  For  example,  operators  still  need  to 

(1)  identify  goals  based  on  their  own  situation  assessment,  (2)  select  the  appropriate  procedure,  (3)  evaluate 
whether  the  procedure-defined  actions  are  sufficient  to  achieve  those  goals,  and  (4)  adapt  the  procedure  to  the 
situation,  if  necessary. 

The  decision  making  involved  in  situation  assessment  and  response  planning,  especially  in  ambiguous  situations 
when  available  procedures  do  not  suffice,  can  be  a  large  cognitive  burden  and  draw  heavily  upon  working  memory, 
long-term  memory,  and  attentional  resources.  Information  then  is  consciously  manipulated  in  working  memory, 
and  the  ability  to  do  so  is  a  direct  function  of  attentional  resources  available.  Working  memory  has  very  limited 
capacity,  and  without  sustained  attentional  resources  (or  transfer  of  the  information  to  long-term  memory), 
information  decays  rapidly.  Information  can  be  lost  due  to  (1)  insufficient  attentional  resources  to  keep  it  active, 

(2)  overload  of  the  working-memory  capacity,  and  (3)  interference  from  other  information  in  working  memory.  To 
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increase  the  capacity  of  working  memory,  operators  use  memory  heuristics,  such  as  chunking  -  aggregating  and 
organizing  information  into  higher-level,  meaningful  units.  A  heuristic,  as  used  here,  means  a  shortcut  for 
information  processing  developed  through  experience  and  trial-and-error,  rather  than  systematic,  formal  analysis. 
Once  this  is  accomplished,  the  higher-level  units,  not  the  individual  elements,  are  stored  in  working  memory. 

Dien  et  al.  (1991)  discussed  the  importance  of  higher-level  cognitive  functions  when  operators  use  procedures. 
Operators  must  compensate  for  inadequacies,  fill  in  gaps,  and  resolve  conflicts  between  the  control  objectives 
specified  in  the  procedures  and  those  established  by  the  operators  assessing  the  situation.  Operators  sometimes 
must  implement  more  practical  strategies  than  those  in  the  procedures.  They  must  also  consider  whether  operating 
actions  should  be  anticipated,  or  whether  automatic  devices  should  be  left  to  operate. 

Roth  et  al.  (1994)  demonstrated  the  need  to  maintain  a  supervisory  role,  even  when  responses  are  largely  dictated 
by  EOPs.  They  investigated  how  operators  handle  cognitively  demanding  emergencies,  their  objective  being  to 
examine  the  role  of  situation  assessment  and  response  planning  on  guiding  a  crew’s  performance  when  EOPs  were 
being  utilized.  NPP  operators  from  two  different  utilities  performed  interfacing  systems  loss-of-coolant  accident 
(ISLOCA)  and  loss-of-heat-sink  scenarios  on  training  simulators  where  complexities  made  it  difficult  to  simply 
follow  the  appropriate  procedure.  The  results  illustrated  the  importance  of  high-level  cognitive  functions  during 
use  of  EOPs.  The  operators  developed  an  understanding  of  the  plant’s  state  and  confirmed  their  situation 
assessment,  and  also  attempted  to  understand  plant  performance  that  was  unexpected  based  on  their  current 
situation  model.  These  cognitive  activities  enabled  them  to  evaluate  the  appropriateness  of  the  EOP  to  achieve  the 
high-level  goal  dictated  by  the  situation  assessment  within  the  context  of  current  conditions.  Roth  et  al.  noted  the 
importance  of  the  crew’s  interactions  and  communications  to  these  high-level  cognitive  functions,  due  partly  to  the 
need  to  obtain  information  from  many  HSIs  in  different  locations.  In  addition,  communication  helped  operators 
overcome  the  fact  that  EOPs  do  not  cover  all  the  important  information  on  the  current  plant  state.  When  a  specific 
procedure  seemed  to  fail  to  meet  the  high-level  goal,  operators  would  alter  the  procedure  path  to  better  address  the 
current  situation. 

Thus,  Roth  et  al.  (1994)  demonstrated  the  importance  of  understanding  the  basis  of  the  procedure  and  its  intended 
higher-level  goals.  The  need  to  formulate  modifications  to  procedure  pathways  also  means  that  operators  may  not 
simply  proceed  linearly  through  a  procedure.  They  may  need  to  consider  future  steps,  reexamine  previous  ones, 
and  refer  to  other  procedures  to  verify  that  their  current  activities  are  correct  and  will  meet  the  high-level  goals. 

Roth  (1994)  considered  the  implications  of  the  Roth  et  al.  (1994)  study  for  designing  operator  support  systems. 
First,  the  requirement  of  situation  assessment  and  response  planning  independent  from  procedures  suggests  that 
operators  must  maintain  awareness  of  abnormal  plant  symptoms,  determine  what  malfunctions  could  produce 
them,  and  know  the  manual  and  system  actions  that  are  being  undertaken  and  their  effects.  'Second,  since  crews 
must  anticipate  the  consequences  of  their  actions,  operator  support  systems  could  help  in  identifying  their 
consequences  and  side  effects.  Third,  operators  must  understand  the  assumptions  and  logic  behind  the  procedures, 
i.e.,  their  intent,  their  overall  strategies,  and  the  transition  logic  between  them.  Since  operators  may  not  move 
linearly  between  procedures,  CBP  navigation  systems  will  be  imp)ortant  to  the  success  of  CBPs  in  complex 
emergencies. 

Monitoring  and  Detection 

Monitoring  and  detection  refer  to  the  activities  involved  in  extracting  information  from  the  environment. 
Monitoring  is  checking  the  state  of  the  plant  to  determine  whether  the  systems  are  operating  correctly;  it  can 
include  checking  parameters  indicated  on  the  CR  panels,  monitoring  those  displayed  by  the  process  computer, 
obtaining  verbal  reports  from  operators  in  the  plant  areas,  and  sending  them  to  other  areas  to  check  equipment. 
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Detection  is  the  operator's  recognition  that  something  is  operating  abnormally.  Procedures  guide  monitoring  and 
detection  by  identifying  the  parameters  to  be  monitored.  However,  operators  must  also  monitor  the  crew’s  actions 
taken  in  response  to  disturbances. 

Monitoring  and  detection  are  influenced  by  two  factors:  (1)  the  characteristics  of  the  environment,  and  (2)  the 
operator's  knowledge  and  expectations.  These  factors  lead  to  two  types  of  monitoring:  (1)  data-driven,  and 
(2)  model-driven.  Monitoring  driven  by  environmental  characteristics  often  is  referred  to  as  data-driven 
monitoring.  Data-driven  monitoring  is  affected  by  the  salience  of  the  information’s  presentation  (e.g.,  size,  color, 
and  loudness).  Thus,  alarm  systems  are  basically  automated  monitors  designed  to  influence  data-driven 
monitoring  by  using  physical  salience  to  attract  attention.  Auditory  alerts,  flashing,  and  color  coding  are  physical 
characteristics  that  enable  operators  to  quickly  identify  an  important  new  alarm.  Data-driven  monitoring  also  is 
influenced  by  the  behavior  of  the  information,  such  as  the  bandwidth  and  rate  of  change  of  the  information  signal; 
observers  more  frequently  monitor  a  rapidly  changing  signal. 

Operators  may  initiate  monitoring  based  on  their  knowledge  and  expectations  (model-driven)  about  the  most 
important  information;  this  typically  is  called  knowledge-  or  model-driven  monitoring.  Model-driven  monitoring 
can  be  viewed  as  active  monitoring,  in  that  the  operator  is  not  merely  responding  to  environmental  characteristics 
that  ‘‘shout  out”  like  an  alarm  system  does,  but  is  deliberately  directing  attention  to  areas  expected  to  provide 
specific  information. 

Model-driven  monitoring  may  be  initiated  by  several  factors.  First,  it  may  be  guided  by  operating  procedures  or 
standard  practice  (e.g.,  control  panel  walk-downs  at  shift  turnovers).  Second,  it  can  be  triggered  by  situation 
assessment  or  response  planning  activities  and  is,  therefore,  strongly  influenced  by  a  person’s  current  situation 
model.  The  situation  model  allows  the  operator  to  direct  attention  and  focus  monitoring  effectively.  However, 
model-driven  monitoring  can  lead  operators  to  miss  important  information.  For  example,  an  incorrect  situation 
model  may  focus  operators’  attention  in  the  wrong  place,  cause  them  to  fail  to  observe  a  critical  finding,  or  to 
misinterpret  or  discount  an  indication. 

An  operator  is  faced  with  an  information  environment  containing  more  variables  than  can  be  realistically 
monitored.  The  real  challenge  comes  from  the  fact  that  there  are  many  potentially  relevant  things  to  attend  to  at 
any  time,  and  the  operator  must  determine  what  information  is  worth  pursuing  within  a  constantly  changing 
environment  (Vicente,  Mumaw,  and  Roth,  1997).  Then,  the  operator  must  decide  what  to  monitor  and  when  to 
shift  attention  elsewhere.  These  decisions  are  strongly  influenced  by  an  operator’s  current  situation  model,  which 
guides  the  allocation  of  attentional  resources  to  sampling  data  from  the  environment  based  on  its  statistical 
properties;  i.e.,  expected  probability  and  correlation.  The  operator’s  ability  to  develop  and  effectively  use 
knowledge  to  guide  monitoring  relies  on  the  ability  to  understand  the  current  state  of  the  process.  As  cognitive 
workload  increases,  monitoring  strategies  become  less  thorough,  and  the  capability  to  detect  particular  failures 
decreases  (Ephrath  and  Young,  1981). 

Under  normal  conditions,  situation  assessment  is  attained  by  mapping  the  information  obtained  from  monitoring  to 
elements  in  the  situation  model.  For  experienced  operators,  this  comparison  is  relatively  effortless.  During 
unfamiliar  conditions  the  process  is  considerably  more  complex.  The  first  step  in  realizing  that  the  current  plant 
conditions  are  inconsistent  with  the  situation  model  is  detecting  a  discrepancy  between  information  representing 
the  current  situation  and  that  derived  from  monitoring.  This  process  is  facilitated  by  the  alarm  system,  which 
directs  the  operator’s  attention  to  an  off-normal  situation. 

When  determining  whether  a  signal  is  significant  and  warrants  further  investigation,  operators  examine  it  in  the 
context  of  their  current  situation  model.  They  must  judge  whether  the  anomaly  indicates  a  real  abnormality  or  an 
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instrumentation  failure.  They  then  will  assess  the  likely  cause  of  the  abnormality,  and  evaluate  the  importance  of 
the  signal  in  determining  their  next  action. 

Monitoring  has  been  described  in  terms  of  signal  detection  theory  (SDT)  (Green  and  Swets,  1988).  Process  control 
operators  are  in  a  monitoring  environment  described  in  SDT  terms  as  an  alerted-monitor  system  (Sorkin  et  al., 
1985;  Sorkin  et  al.,  1988).  Such  a  system  is  composed  of  an  automated  monitor  and  a  human  monitor.  The 
automated  monitor  in  an  NPP  is  the  alarm  system  which  detects  off-normal  conditions.  When  a  parameter  exceeds 
the  criterion  of  the  automated  monitor,  the  human  monitor  is  alerted  and  then  must  detect,  analyze,  and  interpret 
the  signal  as  a  false  alarm  or  a  true  indication  of  an  upset.  The  human  monitor  also  can  assess  plant  parameters 
independently  of  the  automated  monitor  (the  alarm  system).  Both  monitors  have  their  own  specific  detection 
parameter  values  for  sensitivity  (d')  and  response  criterion.  The  latter  refers  to  the  amount  of  evidence  needed 
before  an  operator  will  conclude  that  a  signaled  event  is  actually  present;  this  is  called  response  bias  since  it 
describes  an  operator’s  conservatism.  Sensitivity  refers  to  the  resolution  of  the  system,  i.e.,  the  ease  with  which 
signals  (represented  as  a  statistical  distribution)  can  be  distinguished  from  signals  plus  noise  (similarly 
represented). 

SDT  research  has  many  implications  for  understanding  how  operators  process  information  during  a  disturbance. 
First,  the  response  criterion  is  affected  by  expectancy;  i.e.,  the  expected  probability  that  an  event  will  occur  and  the 
payoff  structure  (rewards  and  penalties  for  making  correct  and  incorrect  detections,  respectively).  While  alarms 
can  occur  frequently,  significant  off-normal  events  in  NPPs  typically  have  a  low  probability.  Therefore,  operators 
have  low  expectancy  about  their  actual  occurrence  which  creates  a  conflict  between  the  cost  to  productivity  for 
falsely  taking  an  action  that  shuts  down  the  reactor  versus  the  cost  for  failing  to  take  a  warranted  action.  In 
actuality,  because  disturbances  have  a  low  probability,  operators  rely  on  redundant,  supplemental  information  to 
confirm  the  alarm.  Having  verified  several  confirmatory  indicators,  the  operator  can  accept  the  alarm  information 
as  indicating  an  actual  off-normal  condition. 

There  are  two  types  of  anomalies:  (1)  deviations  from  desired  system  functions,  called  abnormal  findings,  and 
(2)  deviations  from  the  operator’s  situation  model,  or  unexpected  findings.  The  two  anomalies  lead  to  different 
follow-up  reasoning  and  monitoring  behavior: 

•  Abnormal  findings  lead  to  information  processing  about  how  to  cope  with  the  disturbance  (response  planning) 
and  to  monitoring  behavior  to  see  if  the  expected  coping  responses  have  occurred  and  are  having  the  desired 
effect. 

•  Unexpected  findings  or  process  behavior  lead  to  situation  assessment  activity  and  model-driven  monitoring  to 
explain  the  finding. 

Failures  in  monitoring  can  include  failing  to  observe  parameters,  misunderstanding  the  significance  of  parameters, 
or  failing  to  obtain  needed  reports  from  plant  areas.  Failures  in  detection  can  include  failure  to  recognize  an 
abnormality  despite  proper  monitoring.  An  error  in  monitoring  or  detection  can  cause  the  operator’s  failure  to 
respond  to  the  event,  or  at  least,  to  respond  within  the  required  time. 

Response  Implementation 

Response  implementation  is  the  performance  of  the  actions  identified  in  response  planning.  This  can  be  as  simple 
as  an  individual  operator  selecting  and  operating  a  control,  or  it  can  involve  communications  and  coordination 
with  teams  of  operators  in  different  parts  of  the  plant,  who  each  then  select  and  operate  equipment  controls  in  a 
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centrally  coordinated  manner.  The  actions  may  be  discrete  (e.g.,  flipping  a  switch)  or  may  involve  continuous 
control  (e.g,,  controlling  steam  generator  level). 

The  results  of  actions  are  monitored  through  feedback  loops.  Two  aspects  of  NPPs  jeopardize  the  implementation 
of  responses:  time  and  indirect  observation.  Time  and  feedback  delays  disrupt  response  implementation  because 
they  make  it  difficult  to  determine  whether  control  actions  are  having  their  intended  effect.  Consequently,  the 
operator's  ability  to  predict  future  states  from  mental  models  can  be  more  important  in  controlling  responses  than 
feedback.  Further,  since  plant  processes  cannot  be  directly  observed,  their  status  is  inferred  through  indications; 
thus,  errors  in  cognition  can  impede  performance. 

Failures  in  implementing  a  response  can  lead  to  the  operation  of  the  wrong  equipment,  or  the  incorrect  operation 
or  control  of  particular  components. 

Summary 

The  role  of  the  operating  crew  in  an  NPP  is  that  of  a  supervisory  controller  that  must  engage  in  situation 
assessment,  monitoring  and  detection,  and  response  planning  and  implementation.  These  cognitive  functions  are 
applied  to  tasks  for  which  the  crew  has  primary  responsibility,  as  well  as  to  automated  systems  and  systems 
designed  to  support  crew  tasks.  Procedures  fall  into  the  latter  category.  Historically,  procedures  were  designed  to 
support  response  planning  by  providing  operators  with  strategies  that  were  based  on  previous  detailed  analyses  of 
normal  and  abnormal  plant  states.  However,  when  these  preplanned  strategies  are  applied  to  the  unique 
circumstances  of  a  particular  disturbance,  unforseen  or  unanticipated  situations  may  render  an  aspect  of  a 
procedure  inappropriate  or  ineffective.  Thus,  confronted  with  the  complexities  of  real-world  process  disturbances, 
operators  must  monitor  the  performance  of  the  procedure  to  verify  its  conformity  to  the  higher-level  goals  that  it 
was  designed  to  achieve.  Under  such  circumstances,  it  is  important  for  operators  to  assess  the  effectiveness  of  the 
response  plan  even  when  it  is  described  by  established  procedures,  evaluate  the  consequences  of  particular 
procedure  actions,  and  evaluate  the  appropriateness  of  the  procedure  path  for  achieving  identified  goals.  This 
assessment  enables  operators  to  detect  when  procedures  are  not  achieving  the  goals,  when  they  may  contain  errors, 
or  when  errors  are  made  in  carrying  out  procedure  steps.  Another  cognitive  activity  is  adapting  the  response  plan. 
This  includes  filling  in  gaps  in  a  procedure,  modifying  it  to  fit  the  specific  situation,  redirecting  its  path,  and  using 
additional  or  alternative  procedures. 

Thus,  rather  than  assuming  the  role  of  rote,  verbatim  “procedure-followers,”  it  is  important  that  the  operators 
maintain  the  role  of  supervisory  controllers  and  monitor  the  performance  of  the  procedures  as  well  as  the  process. 

With  the  development  of  CBPs,  the  support  of  procedure  systems  extends  beyond  response  planning  and  includes 
aspects  of  situation  assessment,  monitoring  and  detection,  and  response  implementations.  This  support  may  be 
applied  to  the  operator’s  primary  tasks  as  well  as  secondary  tasks. 

5.2  Issues  with  Paper-Based  Procedures  and  Implications  for 
Computerization 

As  we  discussed,  plant  procedures  provide  instructions  to  guide  operators  in  monitoring,  deciding  on  appropriate 
actions,  and  controlling  the  plant.  The  design  of  procedures  was  identified  as  a  major  cause  of  human  error 
because  PBPs  have  characteristics  limiting  the  manner  in  which  information  can  be  presented,  and  impose  tasks 
upon  operators  that  are  not  directly  related  to  controlling  the  plant.  Properly  following  the  procedures  sometimes 
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is  complicated  by  the  necessity  to  track  several  EOPs  or  EOP  branches  simultaneously.  Current  symptom-based 
procedures  appear  to  place  significant  workloads  on  operators. 

The  purpose  of  examining  the  human  performance  issues  associated  with  PBPs  is  to  (1)  ensure  that  these  design 
deficiencies  are  addressed  by  CBPs  where  possible,  and  (2)  identify  aspects  of  procedure  use  that  can  be  supported 
by  computerization  where  human  performance  issues  are  not  a  simple  function  of  the  paper  medium. 

In  this  section,  the  factors  associated  with  difficulties  with  PBPs  are  identified,  and  their  implications  for  CBP 
design  are  discussed. 

Issues  with  PBPs 


Several  studies  have  addressed  the  problems  associated  with  PBPs,  and  identified  a  broad  range  of  deficiencies  that 
fall  into  the  following  categories:  design  process,  implementation,  training,  and  maintenance.  The  deficiencies 
identified  in  these  studies  came  from  various  sources  including  NRC  procedure  inspections,  operator  interviews, 
and  literature  reviews.  Table  5.1  compiles  the  overall  results.  Four  studies  whose  findings  are  included  in  the 
table  are  briefly  discussed  next. 

For  non-EOP  procedures,  the  types  of  deficiencies  identified  include  (1)  an  excessive  number  of  procedures  and 
poor  classification  schemes  for  their  use,  (2)  technical  inaccuracies,  (3)  lack  of  clearly  specified  goals  and  criteria 
for  determining  that  the  intent  of  the  procedure  was  attained,  and  (4)  vaguely  written  procedures  that  do  not 
specifically  describe  the  necessary  actions  (Morgenstem  et  al.,  1987;  Barnes  and  Radford,  1987). 

These  procedure  limitations  were  associated  with  numerous  problems  in  performance.  Flow  control  and  transitions 
between  procedures  can  be  associated  with  potential  safety-significant  errors  (Chignell  and  Zuberec,  1993)  when 
operators  do  the  following: 

•  Skip  a  step  in  the  procedure 

•  Follow  out  of  sequence 

•  Inadvertently  use  the  wrong  step 

•  Follow  an  out-of-date,  erroneous,  imprecise,  or  ambiguous  procedure 

•  Follow  the  wrong  procedure  due  to  incomplete  procedure  references 

•  Miss  a  procedure  transition  and  continue  in  the  current  one 

•  Become  lost  or  confused  when  a  transition  is  identified  in  a  caution  rather  than  as  an  action  step 

Teamwork  and  communication  are  also  important.  Hoecker  et  al.  (1994)  and  Hoecker  and  Roth  (1996)  identified 
errors  in  communication  during  the  acquisition  of  procedure-specified  information  as  problematic,  including 
delays,  suspended  tasks,  and  difficulty  identifying  the  correct  display  from  other  displays.  These  limitations 
increase  workload  and  the  likelihood  of  procedural  errors. 
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Implications  for  CBPs 

The  deficiencies  identified  in  PBPs  are  associated  with  implementing  the  procedures  as  well  as  with  the  procedure 
design  process,  training,  and  maintenance  (illustrated  in  Table  5.1).  Teigen  and  Ness  (1994)  identified  the 
following  CBP  features  as  important  in  addressing  these  limitations: 

•  A  structured,  consistent  format 

•  Ease  of  transition  between  procedures,  and  recording  transitions 

•  Place  keeping  in  procedures  when  operators  are  in  several  simultaneously  or  when  they  access  support 
information,  such  as  tables  and  charts 

•  Clear,  consistent  logic  statements 

•  Monitoring  of  process  control  parameters 

•  Simplification  of  flowcharts  by  allowing  operators  access  to  varying  levels  of  detail 

•  Maintenance  of  procedures 

Similarly,  Lipner  and  Rusnica  (1996)  identified  some  of  the  CBPs’  features  that  can  reduce  the  mental  load  and 
time  demands  of  working  with  PBPs.  They  included  monitoring  plant  parameters,  centralizing  all  procedural 
information  in  one  place,  creating  detailed  record  keeping  on  procedure  implementation,  and  facilitating  the 
maintenance  of  procedures. 

CBPs  can  directly  affect  many,  but  not  all  such  deficiencies.  In  fact,  a  significant  consideration  in  evaluating  CBPs 
may  be  the  extent  to  which  they  solve  these  problems.  Next,  we  describe  each  of  the  problems  associated  with  the 
paper  medium  and  their  possible  resolutions  by  CBPs.  For  those  problems  not  directly  impacted  by 
computerization,  many  of  the  same  factors  that  contribute  to  PBP  problems  can  also  undermine  CBPs. 

We  now  consider  the  major  areas  of  procedure  use  that  can  be  impacted  by  computerization.  Several  categories 
were  previously  identified  (Barnes  et  al.,  1996).  While  they  mainly  cover  implementation  of  procedures,  an  initial 
category  is  identified  which  includes  the  more  general  areas  of  design  process  and  support: 

•  Design  Process  and  Support 

•  General  Cognitive  Workload 

•  Level  of  Detail 

•  Context  Sensitivity 

•  Sequence  Control  and  Navigation 

•  Management  of  Multiple  Procedures 

•  Maintenance  of  Technical  Accuracy  of  Procedures 

•  Integration  of  Procedure  Tasks  and  Other  Tasks 
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Table  5.1  Deficiencies  in  Paper-Based  Procedures 


Issue 

Deficiency 

Note 

Design 

Process 

•  Inadequate  participation  of  operations  and  training  personnel  in  developing 
procedures 

(1) 

•  Technically  incorrect  EOPs 

(1) 

•  Address  standard  situations,  but  are  less  supportive  in  unusual  situations 

(1) 

•  Incomplete  procedures 

(3) 

•  Inadequate  consideration  of  the  time  required  to  complete  procedural  actions 

(1) 

•  Insufficient  verification  and  validation  (V&V)  of  procedures 

(1) 

Imple- 

•  Non-specific  entry  and  exit  conditions  for  support  procedures 

(1) 

mentation 

•  Fixed  and  inflexible  procedures 

(2) 

•  Incorrect  sequencing  of  action  steps 

(1) 

•  Inadequate  consistency  across  procedure 

(1) 

•  Inconsistencies  in  formatting  and  use  of  terminology 

(1) 

•  Incorrect  identification  of  equipment 

(1) 

•  Inadequate  ability  to  provide  varying  level  of  detail 

(4) 

•  Non-sequential  presentation  of  information 

(4) 

•  Navigation  to  related  information 

(4) 

•  Management  of  multiple  procedures 

(4) 

•  Integration  of  procedure  tasks  and  other  tasks 

(4) 

•  Problems  in  labeling  and  headings 

(3) 

•  Notes  and  cautions  in  improper  places 

(3) 

•  Lack  of  context-dependent  highlighting  and  navigation 

(3) 

•  Need  to  use  multiple  procedures  simultaneously  and  move  between  sections 

(3) 

•  Lack  of  flowcharts  to  guide  use  of  procedure 

(3) 

•  Inadequate  support  and  reference  material 

(1) 

•  Bulkiness 

(3) 

•  Physical  handling  of  procedures  near  control  panels 

(2) 

•  Separation  from  other  information  sources,  such  as  the  safety  parameter 
display  system  (SPDS) 

(3) 

•  Inconsistency  with  other  HSIs  in  referring  to  plant  equipment 

(1) 

Training 

•  Poor  training  of  operators  in  use  of  procedures 

(1) 

Maintenance 

•  Maintaining  technical  accuracy  of  procedures 

(3,4) 

Notes:  (1)  Lapinsky,  1989;  Galletti  and  Sutthoff,  1992 

(2)  Niwa,  Hollnagel,  and  Green,  1996 

(3)  Chignell  and  Zuberec,  1993 

(4)  Barnes,  Desmond,  Moore,  and  O’Hara,  1996 
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Design  Process  and  Support 

The  NRC’s  analysis  of  EOPs  (Lapinsky,  1989)  concluded  that  deficiencies  in  the  design  of  EOPs  were  likely- to  be 
found  when  any  of  the  following  were  lacking: 

•  A  multidisciplinary  team 

•  An  independent  review  to  assure  technical  accuracy  and  usability 

•  A  systematic  process  to  ensure  EOPs  do  not  degrade  over  time 

•  Management’s  commitment  to  the  EOP  design  process 

These  factors  can  also  negatively  impact  the  development  and  use  of  CBPs.  A  development  program  should 
address  these  issues  and  those  in  the  non-implementation  categories  in  Table  5.1  to  ensure  that  the  final  CBP 
system  will  adequately  support  safe  operations. 

General  Cognitive  Workload 

Many  problems  with  PBPs  result  from  the  high  demands  on  cognitive  activities,  especially  monitoring  (e.g.,  of 
parameters  needed  to  use  procedure  logic),  decision  making  (e.g.,  analyzing  procedure  step  logic),  and  memory 
(e.g.,  to  perform  steps  of  continuous  applicability).  These  problems  are  amplified  by  the  stress  created  by  complex 
process  disturbances.  Mumaw  (1994)  found  that  stress  lowers  human  performance  by  (I )  narrowing  and  shifting 
attentional  focus,  (2)  reducing  working  memory  capacity,  and  (3)  impairing  the  crew’s  communication  patterns. 
Reducing  the  demands  on  cognitive  processes  can  support  the  operator  in  managing  stress  and  maintaining 
performance.  By  supporting  cognitive  functions  such  as  obtaining  parameter  values  (monitoring),  comparing  them 
to  reference  values,  and  monitoring  steps  of  continuous  applicability  (discussed  further  below),  CBPs  may  reduce 
the  demands  on  attentional  resources  and  working  memory,  enabling  operators  to  focus  on  evaluating  higher-level 
procedure  goals. 

Level  of  Detail 

Space  for  explanatory  information  is  limited  in  PBPs  and  the  level  of  detail  in  procedure  steps  is  fixed. 

Determining  the  appropriate  level  of  detail  in  presenting  procedure  steps,  and  deciding  upon  the  type  and  level  of 
detail  in  supporting  information  are  inexact  processes  that  may  be  facilitated  by  computerization.  Substantial 
interaction  with  trainers  and  operators  is  needed  to  decide  upon  the  level  of  detail,  which  increases  costs  and  still 
has  an  uncertain  outcome.  Linkages  between  training  programs  and  the  procedures  can  lessen  over  time,  and  so 
the  operators'  knowledge  bases  can  change.  In  addition,  even  though  all  operators  have  reached  the  expertise 
required  for  licensing,  differential  experience  may  lead  to  differences  in  familiarity  with  the  components,  systems, 
and  processes  defined  in  the  procedures.  Those  more  familiar  with  a  task  may  become  impatient  if  the  information 
is  too  detailed  and  may  inadvertently  skip  steps  to  avoid  wading  through  unneeded  information.  Operators  who  are 
less  familiar  with  the  task  may  be  unable  to  perform  the  procedure  correctly  with  the  amount  of  information 
provided. 

An  advantage  of  CBPs  is  that  they  can  provide  varying  levels  of  detail  (Fischer  et  al.,  1991;  Jenkinson  et  al.,  1991), 
and  computerization  has  been  used  elsewhere  to  resolve  these  problems.  For  example,  in  the  early  1980s,  the  Navy 
developed  computer-based  troubleshooting  aids  for  maintenance  that  allowed  the  user  to  choose  between  two 
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different  levels  of  detail.  In  the  more  detailed  option,  general  steps  (e.g.,  “start  the  pump”)  were  broken  up  into 
specific  substeps  (e.g.,  “depress  the  start  button  on  Pump  XYZ”)  and  more  detailed  graphical  displays  were  shown. 

Hypertext  applications  are  under  development  within  the  Department  of  Defense  (DOD)  as  part  of  the  Continuous 
Acquisition  and  Life-Cycle  Support  (CALS)  initiative  that  also  allow  the  user  to  determine  the  level  of  detail  of 
information.  Some  systems  allow  the  user  not  only  to  set  a  preferred  level  of  detail,  but  also  to  browse  through 
supplementary  information,  and  the  information  that  was  used  to  develop  the  technical  content  of  the  procedure 
steps.  Access  to  this  latter  information  may  be  particularly  useful  if  the  procedure  cannot  be  performed  under 
existing  plant  conditions  and  must  be  modified. 

NPPs  and  vendors  of  nuclear  steam  supply  systems  have  developed  procedure-basis  documents  that  give  significant 
details  and  insights  into  the  reasons  for  procedure  steps  and  the  logic  for  choosing  parameters,  operations,  and  step 
order;  such  information  does  not  normally  appear  in  procedures  to  avoid  clutter.  In  CBPs  this  information  can  be 
made  available  when  necessary. 

Improper  implementation  of  a  variable  level  of  detail  potentially  may  impair  operators’  performance  if  operators  do 
not  understand  the  level  appropriate  to  their  use  of  the  procedure. 

Context  Sensitivity 

Irrelevant  information  about  conditions  that  do  not  exist  during  a  specific  implementation  of  a  procedure  must 
continuously  be  shown  for  decision  steps  in  PBPs.  Because  nearly  all  procedures  involve  decisions,  operators  may 
have  to  read  several  pages  of  irrelevancies  to  find  the  appropriate  action  steps.  This  can  cause  operators  to  lose 
track  of  their  place  in  the  procedures,  to  miss  important  information,  and  to  delay  their  performance. 

Several  techniques  are  used  in  PBPs  to  present  decision  steps,  some  of  which  require  less  space  than  others.  In 
text-format  procedures,  decision  steps  typically  are  represented  by  Boolean  logic  terminology  (e.g.,  “if  the 
following  conditions  exist,  then  perform  these  actions”).  In  flowchart-format  procedures,  the  antecedent  conditions 
are  typically  presented  in  decision  diamonds,  and  consequent  actions  in  rectangular  symbols  that  are  linked  by  flow 
lines  to  the  decision  diamonds.  Other  formats  are  possible,  such  as  the  two-column,  “response  not  obtained” 
format  used  in  EOPs  for  pressurized  water  reactors. 

These  techniques  also  may  be  used  in  CBP  systems,  but  may  create  similar  paging  problems  on  a  VDU.  However, 
CBPs  can  display  only  the  relevant  action  steps  for  existing  conditions.  For  example,  once  the  operator  has 
evaluated  the  existing  conditions  and  chosen  an  action,  only  the  information  relevant  to  that  action  would  be 
displayed.  Alternatively,  a  CBP  system  could  be  designed  to  evaluate  the  existing  conditions  zind  to  choose  the 
action  for  the  user.  Then,  both  the  full  listing  of  possible  antecedent  conditions  and  the  action  steps  for  non¬ 
existent  conditions  could  be  “hidden”  from  the  user.  Although  there  may  be  some  value  in  having  the  decision 
criteria  and  the  “paths  not  taken”  continuously  available  for  review,  a  system  may  be  designed  to  reduce  the 
amount  of  information  displayed  to  reduce  errors  and  improve  the  efficiency  of  task  performance. 

Sequence  Control  and  Navigation 

In  PBPs,  information  is  presented  sequentially.  However,  as  Roth  (1994)  indicated,  non-sequential  access  to  other 
procedure  information  and  support  materials  may  be  necessary  for  operators  to  adequately  assess  the  procedure.  In 
addition,  even  in  current  procedures,  some  steps  are  not  performed  sequentially.  Two  examples  may  clarify  these 
problems  and  show  how  computerization  could  resolve  them.  Steps  of  continuous  applicability  are  performed  at 
any  point  in  a  procedure  at  which  certain  conditions  are  met  (e.g.,  pressurizer  pressure  exceeds  a  given  level). 
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Therefore,  because  the  triggering  conditions  may  be  met  at  any  time,  the  steps  of  continuous  applicability  always 
should  be  immediately  available  to  the  user.  However,  space  on  paper  procedure  pages  is  at  a  minimum  (as  it  is  on 
a  VDU),  so  that  continuously  repeating  the  steps  on  each  page  increases  the  amount  of  skimming  through  pages 
that  the  operator  must  do.  Further,  because  the  steps  of  continuous  applicability  are  not  part  of  the  direct  sequence 
of  actions  means  that  they  can  be  easily  overlooked,  even  if  they  are  invariably  shown  in  a  dedicated  box  on  the 
procedure  page  or  on  a  facing  page.  If  a  CBP  system  was  designed  to  detect  the  triggering  conditions  for  these 
steps,  it  could  insert  the  appropriate  step  exactly  when  it  is  needed,  so  that  the  operator  could  immediately  attend  to 
it.  Thus,  space  on  the  VDU  screen  is  not  wasted  with  a  continuous  display. 

Time-dep)endent  steps  are  similarly  problematic  because  they  are  only  performed  after  some  specified  period  (e.g., 
some  NRC  notification  requirements  in  EOPs).  Because  the  time  taken  to  progress  through  a  procedure  may  vary 
under  different  circumstances,  it  is  difficult  to  show  a  time-dependent  step  exactly  when  in  the  sequence  of  steps  it 
must  be  undertaken.  Presenting  the  time-dependent  step  at  the  point  in  the  sequence  that  “starts  the  clock”  may 
mean  that  operators  forget  to  perform  it  after  the  designated  amount  of  time  hcis  passed.  Continuously  displayed 
reminders  have  the  same  limitations  as  repeating  steps  of  continuous  applicability  in  PBPs.  However,  because 
timekeeping  is  easily  automated,  a  CBP  system  could  have  the  timekeeping  function  (where  either  the  operator  or 
the  system  “starts  the  clock”)  and  then  display  the  action  step  when  it  must  be  performed. 

In  paper  procedures,  cross-referencing  between  steps  and  procedures  introduces  errors  and  delays  in  task 
performance.  Navigating  through  such  cross-linked  steps  and  procedures  is  a  significant  problem  for  NPP 
operators  with  PBPs  because  cross-references  interrupt  the  user's  sequential  performance  of  action  steps.  For 
example,  unconditional  branches  instruct  the  user  to  leave  the  current  procedure  and  begin  again  in  another 
procedure  or  in  another  section  of  the  same  one.  References  direct  the  user  to  another  procedure  for  supplementary 
information  or  for  a  series  of  action  steps,  after  which  the  user  is  redirected  to  the  original  procedure  and  continues 
then  to  follow  it.  These  non-sequential  movements  through  PBPs  cause  the  operators  to  lose  track  of  their  place  in 
the  original  procedure,  or  to  waste  time  trying  to  locate  the  procedure  to  which  they  are  referred. 

CBP  systems  could  be  designed  to  assist  operators  in  following  cross-references  or  to  eliminate  the  need  for  them. 

A  CBP  system  can  simplify  the  user's  search  task,  for  example,  with  a  menu  of  procedures  allowing  the  user  to 
choose  the  cross-referenced  procedure,  rather  than  having  to  physically  locate  it  in  a  papier  manual.  Operators  can 
select  the  step  to  begin  performing  actions  in  the  cross-referenced  procedure,  rather  than  scanning  a  document  to 
locate  the  desired  steps.  Windowing  or  some  other  technique  can  support  the  function  of  place  keeping  in  the 
original  procedure  so  that  the  user  can  later  return  to  it.  Checkoff  and  place-keeping  functions  can  be  automated, 
so  that  operators  can  easily  determine  where  they  have  been,  what  steps  were  completed,  and  where  they  left  off  in 
various  procedures. 

More  sophisticated  systems  are  under  development  in  the  DOD’s  CALS  initiative.  Interactive  electronic  technical 
manuals  (lETMs)  for  maintenance  tasks  are  being  developed  in  which  procedure  steps  and  other  elements  are 
stored  in  a  database.  At  the  beginning  of  a  task,  the  user  specifies  the  task  and  the  circumstances  under  which  it 
will  be  performed.  The  sequence  of  action  steps  and  dissociated  supplementary  information  (i.e.,  a  complete 
procedure)  then  are  generated  by  the  system  from  the  database,  so  that  no  cross-referencing  is  required. 

Management  of  Multiple  Procedures 

Physical  management  of  multiple  procedures  and  place  keeping  when  concurrently  carrying  them  out  are  awkward 
with  PBPs;  the  EOPs  currently  used  in  many  boiling  water  reactors  (BWRs)  are  an  example.  Operators  who  must 
use  EOPs  based  on  the  General  Electric  (GE)  Owners'  Group  technical  guidelines,  in  some  emergency  scenarios 
may  have  to  manage  the  concurrent  performance  of  actions  in  up  to  14  different  procedures.  The  bulk  of  the  action 
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steps  are  shown  in  flowcharts  the  size  of  engineering  drawings;  finding  laydown  space  for  these  procedures  in  a 
typical  CR  is  difficult,  and  the  reader  must  physically  move  around  them  to  track  procedure  steps.  In  addition, 
operators  may  be  required  to  carry  out  abnormal  operating  procedures  and  some  system  operating  procedures  at  the 
same  time  as  the  actions  in  the  EOPs.  Prioritizing  the  i^erformance  of  steps  in  the  different  procedures  is  left  to  the 
unit  supervisor,  based  on  the  staff  available  and  the  extent  of  degradation  of  particular  parameters.  The 
opp)ortunities  are  clear  for  the  CR  crew  to  overlook  steps  in  the  procedures  and  to  commit  other  types  of  errors. 

At  a  simple  level,  computer  management  of  the  progress  and  place  keeping  for  multiple  procedures  may  facilitate 
their  use.  At  a  more  sophisticated  level,  CBP  systems  can  be  developed  to  prioritize  and  sequence  the  actions  for 
each  anticipated  emergency  scenario,  and  thereby  substantially  reduce  the  operators’  workload.  However, 
designing  such  a  CBP  system  would  seem  to  necessitate  a  significant  change  in  the  underlying  mitigation  strategy 
adopted  by  the  GE  Owners’  Group.  Because  the  Westinghouse  Owners’  Group’s  Emergency  Response  Guidelines 
already  prioritize  and  sequence  actions  for  pressurized  water  reactor  (PWR)  crews  and  involve  fewer  instances  in 
which  multiple  procedures  must  be  performed,  EOPs  for  PWRs  may  be  easier  to  translate  into  CBPs.  Additionally, 
physical  management  of  PBP  manuals  at  remote  locations,  such  as  the  remote  shutdown  panel,  can  be  difficult. 
CBPs  offer  a  simple  solution  to  this  problem,  as  they  can  provide  the  operators  with  the  same  interface  normally 
used  in  the  main  CR. 

Maintaining  Technical  Accuracy  of  Procedures 

It  can  be  difficult  to  maintain  the  accuracy  of  procedures,  due  to  procedure  modifications  or  to  changes  in  other 
plant  operations  (e.g.,  regulatory  requirements,  equipment  modifications).  Maintaining  technical  accuracy  is 
particularly  difficult  on  paper.  Thus,  a  design  change  in  a  single  component  can  invalidate  every  procedure  that 
refers  to  that  component.  Similarly,  a  procedure  revision  that  changes  the  step  numbers  in  one  procedure  can 
invalidate  every  step  in  other  procedures  cross-referencing  the  changed  procedure.  Some  licensees  already  have 
developed  elaborate  configuration  control  software  to  solve  this  problem. 

Procedure-generation  systems,  such  as  the  lETMs  described  above,  can  overcome  some  of  these  difficulties.  Where 
procedure  actions  and  the  objects  of  those  actions  are  stored  in  a  database  as  objects,  a  change  in  a  step  or  a  part  of 
a  step  in  the  database  will  ensure  that  the  step  is  correct  whenever  it  is  used  in  any  future  procedure  generated  by 
the  system. 

Integration  of  Procedures  and  Other  Tasks 

The  tasks  associated  with  handling  and  reading  a  paper  procedure  may  be  incompatible  with  other  tasks  the 
ojDerator  has  to  perform.  CBP  systems  for  tasks  {performed  at  workstations,  where  the  control  actions  can  be 
jDerformed  at  the  same  workstation  at  which  the  user  obtains  procedures  information,  can  decrease  the  delays  and 
potential  errors  associated  with  PBPs.  Indeed,  many  licensees  assign  the  task  of  reading  the  procedure  to  a 
different  person  to  avoid  delays  and  potential  errors;  CBP  systems  can  eliminate  this  requirement. 

Summary 

Numerous  limitations  of  PBPs  have  been  identified  and  associated  with  delayed  performance  and  human  errors. 
CBPs  offer  the  opportunity  to  rectify  these  problems.  The  following  are  PBP  issues  and  the  types  of  CBP  support 
that  may  address  them: 

•  Design  Process  and  Support 
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-  Weaknesses  in  the  design  process  and  management  support  have  led  to  problems  in  PBPs  and  should  be 
addressed  when  developing  CBPs. 

General  Cognitive  Workload 

-  CBPs  may  reduce  the  demands  on  attentional  resources  and  working  memory,  and  enable  the  operator  to 
focus  more  on  evaluating  higher- level  procedure  goals. 

-  CBPs  can  support  cognitive  functions,  such  as  obtaining  parameter  values  (monitoring),  comparing  actual 
values  to  reference  values  (resolution  of  procedure  step  logic),  and  monitoring  steps  of  continuous 
applicability. 

Level  of  Detail 

-  CBPs  can  allow  adjustment  of  the  level  of  detail  for  operators  with  varying  familiarity  with  the  tasks, 
components,  systems,  and  processes  defined  in  the  procedures  may  enable  them  to  use  procedural 
guidance  more  efficiently. 

Context  Sensitivity 

-  CBPs  can  display  only  the  relevant  procedure  steps  for  existing  conditions,  so  operators  are  not  distracted 
by  irrelevant  information. 

Sequence  Control  and  Navigation 

-  CBPs  can  take  advantage  of  non-sequential  access  to  information  using  computer  navigation  functions. 

-  CBPs  can  automatically  detect  the  triggers  for  the  steps  of  continuous  applicability  and  time-dependent 
steps  and  insert  the  action  step  to  be  performed  exactly  when  it  is  needed. 

-  CBPs  can  perform  cross-referencing,  place  keeping,  and  checkoffs. 

Management  of  Multiple  Procedures 

-  CBPs  can  manage  the  progress  of,  and  place  keeping  for,  multiple  procedures. 

-  CBPs  can  prioritize  and  sequence  the  needed  actions. 

-  CBPs  can  eliminate  some  physical  problems  and  coordinate  many  procedure  manuals,  especially  when 
laydown  space  is  unavailable. 

Maintaining  Technical  Accuracy  of  Procedures 

-  Procedure-generation  systems  using  procedure  databases  can  enhance  accuracy. 

Integration  of  Procedures  and  Other  Tasks 
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-  Control  actions  for  executing  the  procedure  can  be  performed  at  the  same  workstation  as  the  CBP,  and  can 
decrease  the  delays  and  the  potential  for  errors  associated  with  using  PBPs. 

We  note  that  some  of  these  concepts  have  not  yet  been  fully  implemented  or  evaluated.  Others  may  require 
substantial  changes  in  the  manner  in  which  technical  information  is  managed  in  NPPs. 


5.3  Existing  Guidance  for  CBP  Systems 

In  this  section,  we  discuss  the  available  guidance  for  CBP  systems.  Three  principal  conclusions  emerge.  First, 
guidance  for  CBP  systems  is  extremely  limited.  Wourms  and  Rankin  (1994)  noted  that  no  comprehensive 
standards  or  guidelines  are  available  for  designing  or  evaluating  CBPs.  EPRI  (1991)  indicated  that  “...  guidelines 
for  such  soft  procedures  are  not  well  established  and  will  have  to  be  developed  by  the  M-MIS  designer.” 

Second,  there  is  some  uncertainty  over  generalizing  the  principles  and  guidelines  for  PBPs  to  CBPs.  Converse 
(1992)  stated  that  “...  There  is  no  evidence  that  guidelines  for  the  design  of  traditional  hard-copy  procedures  can  be 
successfully  applied  to  computerized  procedures,  and  few  guidelines  specifically  address  the  design  of 
computerized  procedures”  (p.  170).  Similarly,  Tolbert  et  al.  (1991)  concluded  while  much  is  known  about  the 
design  of  PBPs,  the  applicability  of  the  information  to  CBPs  is  unknown.  They  also  believe  that  problems  may 
result  from  combining  the  use  of  PBPs  and  CBPs  due  to  the  need  to  train  operators  on  both  systems,  to  changes  in 
“allocation  of  function”  due  to  the  CBP,  consistency  differences,  and  issues  of  procedure  maintenance. 

Based  on  the  generally  acknowledged  lack  of  HFE  CBP  guidance,  the  third  conclusion  is  that  the  development  of 
CBP  systems  for  operational  use  should  proceed  in  a  way  such  that  the  benefits  and  drawbacks  of  CBP  systems  can 
be  fully  evaluated  for  each  specific  system.  From  reviewing  the  literature  on  CBPs,  Chignell  and  Zuberec  (1993) 
determined  that  “...  a  cautious  approach  should  be  taken  [to  computerization  of  procedures].  Relatively  little  is 
known  about  how  operating  procedures  should  be  used  in  practice,  and  there  is  a  possibility  that  problems  with 
existing  hard  copy  procedures  may  be  compounded  when  they  are  computerized”  (p.  1).  Similarly,  after  reviewing 
several  CBP  systems,  Spurgin,  Wachtel,  and  Moieni  (1993)  concluded  that  “...  more  work  needs  to  be  done  before 
the  industry  can  make  a  safe  transition  from  traditional  paper  and  pencil  procedures  to  computerized  systems” 

(p.  1017). 

In  the  remainder  of  this  section,  the  existing  sources  for  high-level  CBP  design  principles  and  guidance  are 
discussed. 

EdF  CBP  Design  Principles 

Based  on  their  experience  with  developing  CBPs  for  the  N4  design,  EdF  (Dien,  Montmayeul,  and  Beltranda,  1991) 
offered  the  following  general  high-level  guidance: 

•  The  CBP  should  leave  the  operators  in-the-Ioop.  Therefore,  it  has  no  advisory  role  and  leaves  final  decisions 
to  the  operators. 

•  The  CBP  display  screens  should  associate  the  control  objectives,  the  current  process  solution,  and  the  required 
actions. 

•  Operators  should  be  able  to  navigate  freely  within  the  procedure  to  make  up  for  its  insufficiencies. 
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•  The  procedures  should  be  represented  at  different  levels  to  accommodate  various  operator  skill  levels.  The 
EdF  system  has  an  action  level,  which  covers  the  detailed  tasks,  and  an  “objectives-task”  level  showing  the 
chronology  and  links  between  different  objectives. 

It  is  noteworthy  that  several  of  these  guidelines  relate  to  the  postulated  ways  in  which  CBPs  can  improve  on  PBP 
limitations  discussed  previously. 

EPRl  Utility  Requirements  Document 

High-level  CBP  guidance  was  developed  also  by  EPRI  in  the  URD  (EPRI,  1993b).  It  was  based  on  EdFs  CBP 
experience.  EPRI  required  CBP  systems  in  advanced  light  water  reactors  (paragraph  3. 4.2.2).  However,  due  to  the 
lack  of  industry  experience  with  them,  the  URD  suggests  that  simulations  are  needed  to  develop  detailed  guidance 
and  to  validate  the  systems.  The  URD  guidance  is  summarized  below: 

•  Procedures  shall  be  in  the  form  of  logic  or  flowcharts. 

•  The  procedures  shall  normally  provide,  on  the  same  display,  the  parameters  necessary  for  the  operator  to  make 
each  decision. 

•  Plant  parameters  and  status  in  the  procedures  should  be  continuously  updated. 

•  The  operator  will  be  able  to  access  the  control  needed  to  carry  out  the  tasks  directly  from  the  procedure. 

•  The  procedures  should  have  software  to  verify  the  operators’  decisions.  The  operator  shall  retain  control  and 
be  the  final  authority  as  to  whether  or  not  to  proceed.  Disagreements  should  be  automatically  logged. 

•  Where  appropriate,  the  procedures  shall  provide  software  which  retraces  certain  sequences  of  steps  to  assure 
that  proper  status  of  systems  or  components  is  maintained.  These  steps  shall  not  include  actions  taken  by  the 
operators  to  control  components. 

•  For  control  stations  where  CBPs  are  impractical,  and  to  supplement  CBPs,  hard-copy  procedures  should  be 
available. 

•  The  format  and  content  of  hard-copy  procedures  should  be  consistent  with  the  CBPs.  Their  practices  shall 
also  consider  using  PBPs  when  the  normal  CBPs  are  not  available.  In  their  rationale  for  this  guideline,  EPRI 
noted  that  the  correspondence  of  CBPs  and  PBPs  is  important  in  minimizing  the  training  burden  and  the 
potential  for  errors  and  misunderstanding.  Further,  this  consideration  is  especially  important  when  hard-copy 
procedures  are  used  as  a  backup. 

•  The  M-MIS  design  process  shall  include  validation  of  each  operating  procedure  using  the  plant’s  simulator 
and  performance  model. 

The  NRC's  review  of  the  URD  raised  questions  about  the  basis  for  the  last  URD  requirement  (see  RAI  620. 1 3,  in 
NRC,  1991,  pp.  6-7).  As  noted  above,  EPRI  (1991)  indicated  that  CBP  guidance  is  lacking  and  that  it  will  have  to 
be  develop)ed  by  the  designer,  using  simulation.  The  response  noted  that  “...  Since  both  the  ’soft*  and  ’hard* 
procedures  are  subject  to  the  test  of  active  simulation,  there  will  inherently  be  a  direct  comparison  between  the  ’soft' 
and  the  ’hard'  procedures  as  part  of  the  design  process.  Differences  in  operator  performance  with  the  computer- 
presented  procedures  compared  to  the  conventional  printed  procedures  should  be  evident  from  these  evaluations” 
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(p.  31).  Further,  EPRI  indicated  that  If  the  soft  procedures  are  not  concluded  to  represent  an  improvement 
when  active  simulation  is  attempted,  there  is  a  clear  fail-back  to  hard  copy  procedures”  (p.  30).  This  is  consistent 
with  the  URD  requirements  for  an  unproven  HSI  technology.  The  URD  defines  proven  technology  as  one  which 
has  at  least  three  years  of  documented  satisfactory  service  in  a  light  water  reactor  (LWR)  or  similar  application. 
When  these  criteria  cannot  be  met,  a  testing  and  V&V  program  must  be  conducted. 

In  considering  the  EPRI  URD  and  the  subsequent  response  to  the  RAl,  the  staff  noted  the  following: 

...the  development  of  electronically  displayed  procedures  is  a  desirable  goal  for  the  overall 
integration  of  operator  information  needs.  The  staff  position  is  that  the  M-MIS  designer  should 
consider  the  use  of  electronically  displayed  procedures  early  in  the  design  process  to  resolve  any 
issues  concerning  their  development,  operability,  maintainability,  and  reliability.  If  electronically 
displayed  procedures  are  determined  to  be  an  improvement  over  hard-copy  procedures  and  the  M- 
MIS  designer  has  integrated  electronically  displayed  procedures  into  the  overall  M-MIS  design,  they 
should  be  provided  as  part  of  the  design.  (NRC,  1994,  p.  lO.B-17) 

Barnes.  Desmond.  Moore,  and  O'Hara 

Barnes  et  al.  (1996)  developed  a  set  of  principles  representing  a  logical  extension  of  PBP  guidance  to  CBPs  based 
on  their  experience  with  PBP  design,  operational  experience,  and  with  issues  on  computer-based  systems. 
Recognizing  concerns  about  generalizing  PBP  principles  to  CBP  applications,  it  was  done  carefully,  in  a  limited 
fashion. 

The  guidance  was  divided  into  two  primary  sections.  The  first,  “Design  Development,”  covered  aspects  of  the 
design  process  in  procedure  development.  The  second  section,  “Implementation,”  addressed  the  detailed  design  of 
the  procedures  in  the  HSI.  Each  section  was  divided  into  several  subsections: 

•  Design  Development 

-  Concept  of  Operations 

-  Procedure  Bases 

-  Design  Process  Considerations 
Maintainability  of  Computer-Based  Procedures 

-  Training  Specifications 

•  Implementation 

-  General  Considerations 

-  Detailed  Considerations 

Detailed  Interface  Design  for  Constructing  Basic  Steps 
Steps  Containing  Conditional  Relationships 


NUREG/CR-6634 


5-20 


5  TECHNICAL  BASIS  DEVELOPMENT 


Warnings,  Cautions,  and  Notes 
Level  of  Detail 
Organization 

These  guidelines  were  evaluated  and  used  in  the  context  of  the  current  review. 

Niwa.  HollnageK  and  Green 

Niwa,  Hollnagel,  and  Green  (1996)  identified  some  high-level  requirements  for  CBP  systems.  In  general,  they 
stated  that  computerization  should  make  procedure  tasks  easier  without  imposing  additional  tasks,  and  that  CBP 
systems  should  provide  the  following  to  improve  the  use  of  EOPs: 

•  Formatting  -  The  CBPs  can  help  to  structure  the  various  procedure  components  such  as  steps,  conditions, 
comments,  and  advice.  Graphical  techniques  may  also  help  operators  to  understand  the  logical  relationships 
(conditionals,  conjunctive,  and  disjunctive)  defined  in  procedure  steps. 

•  Process  linking  -  Integrating  parameter  values  into  procedure  steps  will  facilitate  their  usage. 

•  Navigation  Support  -  CBPs  can  assist  operators  in  moving  between  procedures  and  support  information. 

•  Progress  monitoring  -  CBPs  can  track  what  steps  were  completed.  Check  boxes  can  be  used,  either  manual  or 
automatic,  depending  on  whether  the  CBP  has  the  specific  criteria  and  information  to  determine  whether  a 
step  was  completed.  Completion  also  can  be  time-stamped  to  facilitate  post-hoc  incident  analysis. 

•  Help  and  explanation  -  Information  can  be  provided  to  help  operators  carry  out  procedure  steps.  For  example, 
the  help  facility  could  describe  how  a  control  action  should  be  carried  out.  The  rationale  for  procedure  steps 
could  also  be  explained.  CBP  systems  could  also  permit  variations  in  the  level  of  detail  based  on  operators’ 
experience  and  input. 

•  Procedure  adaptation  -  CBPs  may  facilitate  changing  a  procedure  to  better  meet  the  current  situation. 

Niwa  et  al.  (1996)  stated  that  their  guidelines  are  good  for  general  aspects  of  procedures,  but  lower-level  details 
should  be  developed  with  operations  and  engineering  personnel.  They  stressed  the  need  for  consistency  between 
the  CBPs  and  the  rest  of  the  HSIs  in  characteristics  such  as  colors,  typography,  interaction  methods,  and  input 
devices.  Such  lack  of  integration  is  a  potential  source  of  risk,  and  reduced  reliability  in  performance.  They  noted 
that  lack  of  consistency  may  be  a  problem  with  third-party  CBP  systems. 

Summary 


The  principal  conclusions  of  this  section  are,  first,  that  while  some  guidance  for  CBP  systems  exists,  it  is  limited. 
Second,  there  is  uncertainty  over  the  generalization  of  PBP  principles  and  guidelines  to  CBPs.  Finally,  CBP 
systems  for  operational  use  should  be  developed  in  such  a  way  that  the  benefits  and  drawbacks  for  each  specific 
system  can  be  fully  evaluated. 
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5.4  Research  on  Computer-Based  Procedure  Systems 

Human  performance  concerns  related  to  CBPs  have  been  raised.  For  example,  Wourms  and  Rankin  (1994) 
indicated  that  CBPs  may  exceed  the  processing  and  attentional  limitations  of  operators.  Chignell  and  Zuberec 
(1993)  suggested  that  operators  may  become  disoriented  and  lost,  may  suffer  from  keyhole  effects,  and  may  lose  the 
location  of  information  in  windows  displays.  In  this  section,  the  effects  of  CBPs  on  performance  are  considered 
from  three  perspectives:  empirical  evaluations  of  performance,  analytical  evaluation  of  CBPs,  and  expert  opinion. 

5.4.1  Empirical  Evaluations  of  CBPs  Based  on  Personnel  Performance 

Empirical  evaluations  of  CBP  systems  and  characteristics  provide  the  best  information  upon  which  to  develop 
guidance  for  design  reviews.  Several  empirical  investigations  of  CBPs  have  been  reported,  but  before  discussing 
them,  it  is  important  to  identify  the  criteria  by  which  such  studies  were  evaluated.  A  most  important  consideration 
is  that  the  CBP  studies  provide  a  basis  from  which  conclusions  can  be  generalized  beyond  the  specific  individual 
study.  Therefore,  the  CBP  studies  were  evaluated  within  the  context  of  validation  reviews  where  generalization 
(external  validity)  is  a  primary  consideration. 

O'Hara,  Stubler,  Brown,  and  Higgins  (1997)  discussed  the  detailed  methodological  considerations  for  validating 
complex  human-machine  systems,  and  developed  a  conceptual  approach  that  identified  important  principles  and 
their  relationships.  The  general  concepts  are  concerned  with  (1)  establishing  the  requirements  for  making  a 
logical,  defensible  inference  from  validation  tests  to  predicted  integrated  system  performance  under  actual 
operating  conditions,  and  (2)  identifying  aspects  of  validation  methodology  that  are  important  to  the  inference 
process.  The  technical  basis  for  inference  is  based  upon  four  general  forms  of  validity:  system  representation, 
performance  representation,  test  design,  and  statistical  conclusion. 

Validity  of  system  representation  refers  to  the  degree  to  which  the  tests  include  aspects  of  the  integrated  system 
that  are  important  to  real-world  conditions.  Specifically,  this  validity  is  based  on  the  representativeness  of  the 
system  model,  human-system  interface,  personnel,  and  operational  events.  Inference  is  supported  to  the  extent  that 
important  aspects  of  the  integrated  system  are  represented  with  high  fidelity,  and  to  the  extent  to  which  important 
contributors  to  potential  variability  in  system  performance  were  adequately  sampled.  It  is  especially  important  in 
evaluating  CBP  studies  that  they  were  conducted  in  situations  -  test  scenarios  -  that  reveal  the  complexity  of 
procedure  use,  and  that  the  procedures  were  used  by  professional  operators  trained  in  their  use. 

Validity  of  performance  representation  refers  to  the  completeness  and  representativeness  of  the  performance 
measures.  A  comprehensive,  hierarchal  approach  to  evaluation  guided  by  supervisory  control  theory  may  be  used 
to  specify  important  aspects  of  p)erformance,  ranging  from  the  operators’  cognitive  processes  to  system  functions. 
The  effects  of  CBPs  on  performance  can  stem  from  both  the  technology  itself,  and  its  interaction  with  the  other  CR 
technologies.  In  general,  the  effects  can  be  related  to  (1)  personnel  role  -  a  change  in  functions  and  responsibilities 
of  personnel,  (2)  primary  tasks  -  a  change  in  the  way  that  personnel  perform  their  primary  tasks,  such  as  process 
monitoring,  situation  assessment,  response  planning,  and  response  implementation  and  control,  (3)  secondary 
tasks  -  a  change  in  the  tasks  the  operator  must  perform  when  interacting  with  the  CBP,  such  as  navigating  through 
displays  and  searching  for  data,  (4)  cognitive  factors  -  e.g.,  a  change  in  cognitive  workload,  and  (5)  personnel 
factors  -  a  change  in  the  required  qualifications  or  training  of  plant  personnel.  The  performance  measures  used 
must  address  these  effects.  Failure  to  include  measures  of  all  important  performance  variables,  poor  measurement 
properties,  and  poorly  specified  criteria  weaken  this  validity  and  the  ability  to  generalize  the  results. 

Validity  of  test  design  addresses  the  procedures  used  for  conducting  the  tests.  Inappropriate  test  procedures  can 
bias  the  relationship  between  the  observations  of  performance  and  the  integrated  system,  and  thus  undermine  their 
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causal  linkage.  When  factors  introduced  by  the  test  methodology  weaken  the  ability  to  interpret  this  correlation, 
validity  is  compromised. 

Finally,  validity  of  statistical  conclusions  addresses  the  relationship  between  the  observed  data  and  established 
performance  criteria. 

While  these  typ>es  of  validity  and  their  associated  methodologies  were  evaluated  in  the  studies  reviewed  in  this 
section,  the  analysis  was  severely  limited  by  the  extent  to  which  the  studies  are  documented.  With  this  caveat  in 
mind,  the  studies  are  discussed  below. 

The  research  is  organized  into  three  sections:  (1)  Comparisons  of  CBP  and  PBP  Systems,  (2)  Observations  of 
Operators’  CBP  Use,  and  (3)  Comparisons  of  CBP  Design  Characteristics. 

Comparisons  of  CBP  and  PBP  Systems 

Spurgin  et  al.  (1990)  compared  the  BWR  Emergency  Op>erating  Procedure  Tracking  System  (EOPTS)  with  PBPs 
in  flowchart  format.  The  study  was  conducted  with  professional  operators  at  the  training  simulator  of  the 
Kuosheng  plant  in  Taiwan.  The  operators  did  not  have  much  experience  with  either  form  of  EOPs.  EOPTS  is 
used  by  the  shift  supervisor.  It  automatically  engages  when  an  entry  condition  is  sp>ecified;  otherwise,  the  EOPTS 
display  screens  are  blank.  The  plant  is  controlled  through  the  normal  HSI. 

The  study  was  conducted  in  two  phases.  In  phase  1,  six  crews  p>erformed  four  scenarios;  three  crews  used  the 
CBPs,  and  three  used  the  PBPs.  The  study  was  undertaken  as  the  crews  were  being  trained  on  EOPTS.  In  phase  2, 
there  were  12  crews,  six  for  each  condition.  The  performance  measures  included: 

•  Time  -  specific  waypoints  were  defined  for  each  scenario,  measured  from  first  cue  to  when  the  appropriate 
response  was  made. 

•  Errors  -  the  number  of  deviations  from  EOPTS-specified  actions  (this  measure  also  was  applied  to  the  PBP 
groups  because,  the  EOPTS  was  running  although  the  crews  could  not  use  it). 

•  System  measures  -  specific  measures  of  system  performance  were  defined  for  each  scenario. 

The  results  showed  that  the  CBP  compared  favorably  to  pap>er  flowcharts.  The  time  measures  for  human 
interactions  were  not  completely  reported.  For  those  that  were,  the  median  response  times  for  the  CBP  crews  were 
faster  in  16  of  the  18  HSIs  analyzed  (an  overall  time  reduction  of  about  75  p>ercent).  In  addition,  the  response 
times  of  CBP  crews  were  less  variable  than  those  of  the  PBP  crews.  There  were  scenario  effects,  as  well.  For  two 
of  the  six  scenarios,  the  response  time  was  slightly  increased  in  the  CBP  group. 

Overall,  the  operators  made  about  twice  as  many  errors  with  the  flowchart  procedures;  unrecovered  errors  were 
65  percent  with  flowcharts,  and  27  percent  using  EOPTS.  When  using  flowcharts,  the  most  likely  source  of  error 
was  misinterpretation  of  a  procedure  statement.  With  EOPTS,  it  was  communication  with  the  control  board 
operators. 

Several  possible  flaws  in  this  study  make  interpretation  of  the  findings  difficult.  First,  since  it  was  a  between- 
groups  design  and  a  small  number  of  crews  was  used,  their  differences  may  have  been  confounded  with  differences 
in  the  presentation  of  procedures. 
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Measurement  was  another  issue.  Errors  were  identified  as  the  number  of  deviations  from  the  specified  path 
through  the  procedure,  as  defined  by  the  CBP.  However,  as  discussed  earlier,  such  a  strategy  of  following 
procedures  by  rote  may  not  be  desirable  (Barnes  et  al.,  1996;  Roth  et  al.,  1994)  and  may  reflect  a  blind, 
unverifiable  approach.  Further,  the  instruction  given  to  the  CBP  group  was  to  follow  the  CBP  verbatim,  but,  no 
equivalent  instructions  were  given  to  the  PBP  group.  Thus,  a  confound  was  created  favoring  the  CBP  group  who 
were  given  the  criterion  for  performance  while  the  PBP  group  was  not.  This  also  affects  the  interpretation  of  the 
“error”  recovery  measure.  Another  issue  was  that  since  the  crew’s  responses  differed  across  crews,  the  appropriate 
procedural  response  differed  within  the  same  scenarios. 

System  measures  were  not  reported  in  any  organized  detail.  In  fact,  they  were  only  reported  for  the  “LOCA  with 
Dry  Well  H2  Control”  scenario.  The  maximum  drywell  hydrogen  concentration  averaged  5.9  percent  for  the  CBP 
group,  and  8.8  percent  for  the  PBP  group.  Further,  the  cumulative  time  below  “top  of  active  fuel”  was  92.5  and 
325  seconds  for  the  CBP  and  PBP  groups,  respectively. 

Without  system  measures  for  other  scenarios,  it  is  difficult  to  assess  whether  the  differences  between  the  crews’ 
performance  on  the  two  procedure  systems  are  meaningful. 

Several  interesting  observations  were  reported  on  situation  awareness  and  crew  communication  and  coordination. 
Spurgin  et  al.  noted  that  SROs  using  EOPTS  were  likely  to  use  it  as  their  primary  way  of  following  a  transient  (i.e., 
not  using  other  HSIs),  which  may  have  hampered  their  awareness  of  the  overall  condition  of  the  plant.  Crew 
members  in  the  EOPTS  condition  who  were  not  using  the  CBPs  expressed  concern  about  being  aware  of  the  EOP’s 
status. 

Time  differences  also  were  important.  One  SRO  called  for  ADS  initiation  twice  because  he  did  not  think  it  had 
occurred  after  his  first  request.  The  misunderstanding  was  due  to  the  delay  in  the  CBP’s  updating  of  ADS  status. 

It  is  interesting  to  examine  the  establishment  of  EOPTS  at  Kuosheng.  The  system  was  gradually  introduced  so  that 
all  plant  personnel  could  become  familiar  with  it.  It  was  first  introduced  into  the  training  simulator  (1)  as  an  aid  to 
instructors  to  track  the  operators’  responses  to  accidents,  (2)  as  a  training  tool  for  the  crews  to  examine  accident 
response  strategies,  and  (3)  as  a  tool  to  be  used  by  the  crews  during  accident  response.  During  this  time,  in 
addition  to  training,  the  correctness  of  EOPTS  was  examined.  Spurgin  et  al.  (1990)  point  out  that  several  errors  in 
the  PBPs  then  were  discovered  as  well.  They  noted  that  crews  in  the  CBP  conditions  operated  much  more  in  the 
skill-  and  rule-based  mode,  while  crews  in  the  PBP  conditions  operated  more  in  the  knowledge-based  mode.  This 
could  certainly  be  another  artifact  of  the  instructions  to  follow  the  procedures  verbatim.  However,  it  is  not 
necessarily  a  positive  outcome  (see  discussion  in  Section  5.1  on  the  need  for  crews  to  maintain  an  independent 
perspective  on  the  procedures).  Spurgin  et  al.  considered  that  the  major  benefit  of  EOPTS  was  that  it  helped 
operators  to  follow  the  procedures  correctly,  and  to  interpret  the  logical  statements  that  are  a  part  of  the  procedure 
steps. 

Another  factor  that  may  have  affected  the  results  was  the  crews’  inexperience  with  either  form  of  procedure.  A 
comparison  of  performance  with  the  flowcharts  between  the  first  and  second  phases  of  the  study  indicated  that 
there  was  a  considerable  improvement.  In  fact,  Spurgin  et  al.  noted  that  one  crew  studied  the  flowcharts  before  the 
scenarios.  They  used  the  process  computer  to  display  important  variables  referenced  in  the  EOPs,  and  their 
response  time  was  comparable  to  that  of  the  EOPTS  crews.  Since  the  CBP  group  was  following  the  CBP  verbatim, 
their  task  may  have  been  easier  only  because  they  did  not  have  to  know  how  to  use  the  flowchart.  If  so, 
performance  differences  between  the  two  groups  may  be  considerably  less  if  they  received  additional  training,  or  if 
they  gained  more  experience  with  the  PBPs. 


NUREG/CR-6634 


5-24 


5  TECHNICAL  BASIS  DEVELOPMENT 


The  observations  of  differences  between  crews  and  scenarios  led  Orvis  and  Spurgin  (1996)  to  recommend  that 
CBPs  should  be  thoroughly  validated  using  several  crews  and  sceneirios.  CBP  systems  should  be  validated  by 
operator-in-the-loop  evaluations  to  ensure  that  they  achieve  their  objectives,  and  that  a  smooth  transition  between 
CBPs  and  PBPs  can  occur  when  necessary. 

In  general,  while  this  study  illustrates  some  potential  benefits  of  CBPs,  the  results  eire  limited  by  (1) 
methodological  confounds  and  procedural  limitations,  (2)  incomplete  reporting  of  data,  (3)  questionable  measures 
of  performance,  and  (4)  underspecification  of  performance  measures,  i.e.,  important  aspects  of  performance  were 
not  measured,  such  as  situation  awareness  and  workload. 

Nelson  et  al.  (1990)  compared  another  procedure  system,  Halden's  Computerized  Operation  Manual  (COPMA),  to 
performance  using  PBPs.  COPMA  was  an  earlier  version  of  COPMA  II.  Fourteen  Halden  reactor  operators 
participated  in  the  study  during  simulated  process  disturbances.  The  dependent  variables  included  time  to  access 
and  complete  the  procedure,  number  of  errors,  and  process  parameters  reflecting  the  operator’s  effectiveness  in 
handling  the  disturbance.  In  general,  it  took  longer  to  access  the  correct  procedure  with  COPMA  than  with  PBPs, 
a  difference  attributed  to  processing  time.  Further,  COPMA  did  not  reduce  the  time  needed  to  perform  procedure 
activities,  and  sometimes  the  PBP  condition  was  significantly  faster.  The  COPMA  group  made  slightly  more 
errors  than  the  PBP  group;  however,  the  differences  were  not  significant.  No  significant  differences  were  observed 
for  the  process  variables. 

On  a  methodological  note,  Folleso,  Meyer,  and  Volden  (1993)  and  Hallbert  and  Meyer  (1995a,  1995b)  indicated 
that  there  were  \arge  differences  between  the  COPMA  and  PBP  groups  as  measured  in  a  pretest,  cind  concluded 
that  the  results  were  confounded  by  them.  When  the  pretest  measures  were  used  to  adjust  the  performance 
measures,  the  differences  between  the  two  groups  lessened  (the  CBP  group  performed  a  little  better  than  the  PBP 
group).  However,  the  assumption  of  confounding  based  on  pretest  differences  may  not  be  warranted  when 
participants  are  randomly  assigned  to  conditions.  Thus,  such  an  approach  for  correcting  the  data  may  not  be 
justified  in  a  randomized  design,  and  is  more  appropriately  used  in  a  quasi-experimental  design  where  non¬ 
equivalent  groups  exist  prior  to  the  study.  Thus,  the  results  for  the  unadjusted  data  are  reported  above. 

Based  on  the  evaluation  of  COPMA,  the  CBP  was  revised  to  produce  COPMA-II.  Some  of  the  changes  were 
increased  functionality,  including  support  for  procedure  search,  improved  instructions  on  procedure  steps,  and 
more  explicit  references  to  procedure  branches. 

Crews’  performance  using  COPMA-II  was  compared  to  PBPs  in  another  study  (Converse,  1994,  1995).  Sixteen 
licensed  operators  managed  a  change  in  power,  small-break  LOCA,  and  a  steam  generator  tube  rupture  on  the 
Scaled  Pressurized  Water  Reactor  Facility  at  North  Carolina  State  University.  The  operators  worked  in  teams  of 
two,  with  an  SRO  managing  the  procedure,  and  a  reactor  operator  (RO)  assisting  in  data  collection  and  control. 

The  dependent  measures  were  procedure  initiation  time,  completion  time,  subjective  estimate  of  workload  using 
the  National  Aeronautics  and  Space  Administration  Task  Load  Index  (NASA-TLX),  and  number  of  errors  (defined 
as  deviations  from  the  “optimal”  sequence  of  procedure  actions). 

The  operators  responded  faster  in  accident  scenarios  with  PBPs,  but  their  response-completion  time  showed  no 
significant  differences.  Measurements  of  the  operators’  accuracy  revealed  an  interaction  between  the  type  of 
procedure  and  the  accident  scenario.  The  error  rate  for  PBPs  was  four  times  higher  than  COPMA-II  for  the  LOCA 
emergency  event,  while  there  was  no  significant  difference  for  the  tube  rupture.  As  with  the  Spurgin  et  al.  study, 
the  meaning  of  the  error  data  may  be  open  to  alternative  interpretation.  Defining  error  as  a  deviation  from  an 
optimal  sequence  may  be  overly  restrictive.  If  an  operator  looks  ahead  at  upcoming  steps,  an  error  is  recorded,  but 
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these  types  of  activities  are  not  necessarily  undesirable  (Roth  et  al.,  1994).  There  were  no  differences  in  workload 
between  the  two  procedures. 

Converse  concluded  that  future  evaluations  of  CBPs  should  systematically  vary  the  type  of  scenario  because  of  the 
different  CBP  effects  on  the  two  types. 

Like  the  EOPTS  study,  the  results  of  this  study  are  difficult  to  interpret.  The  most  significant  result  was  related  to 
error  data  and,  like  the  EOPTS  study,  the  definition  of  error  is  questionable.  No  differences  in  completion  time  nor 
workload  were  found.  Again,  performance  measures  were  underspecified,  i.e.,  situation  awareness  and  plant 
performance  were  not  measured. 

Collier  (1996)  developed  the  following  lessons  learned  from  Malden's  CBP  evaluations: 

•  Operators  must  maintain  an  appropriate  degree  of  control.  The  CBP  system  should  not  overly  structure  the 
operator's  movement  through  the  procedure,  but  should  offer  flexibility  to  skip  steps  or  skim  over  them  quickly 
when  appropriate. 

•  New  MSI  systems  must  offer  an  advantage  over  other  MSI  resources  to  be  used. 

•  To  be  effective,  automation  needs  the  operator's  trust.  One  reason  offered  for  the  slower  performance  with 
COPMA  compared  with  PBPs  was  that  operators  spent  time  double-checking  COPMA  information  because 
they  may  not  have  developed  confidence  in  it. 

Three  other  studies  comparing  CBP  and  PBPs  were  reviewed;  however,  they  did  not  provide  sufficient  information 
to  analyze  them  in  detail.  Kang  (1997)  described  an  intelligent  MSI  being  developed  in  Korea  by  the  Korea  Power 
Engineering  Company,  Inc.,  that  included  a  CBP  for  EOPs.  Each  step  is  composed  of  an  observation,  judgement, 
and  control.  Observations  are  performed  automatically.  The  system  makes  two  types  of  judgements.  Quantitative 
judgements  are  easily  made.  Qualitative  judgements,  for  example,  “If  RCS  average  temperature  is  greater  than 
292®C  and  increasing,  then...”  are  evaluated  by  fuzzy  logic.^  The  level  of  automation  is  varied  depending  on  the 
complexity  of  the  required  decisions.  For  simple  skill-  and  rule-based  tasks,  the  system  is  automated.  For 
knowledge-based  tasks,  control  is  manual.  The  related  piping  and  instrumentation  diagram  (P&ID)  and  summary 
information  are  automatically  presented.  The  system  was  tested  in  a  steam  generator  tube  rupture  scenario  with  a 
full-scope  simulator.  The  response  time  for  completing  the  required  actions  were  compared  with  hard-copy  EOP 
performance.  The  operators  took  37  percent  more  time  with  the  PBP  system  than  with  the  CBP. 

The  Emergency  Operator  Support  System  was  developed  in  Japan  to  support  the  transition  from  event-  to  function- 
based  emergency  procedures,  and  for  using  of  the  EOPs  (Yamamoto  and  Ito,  1993).  The  system  automatically 
displays  the  highest  priority  procedure  in  a  flowchart  form  consistent  with  that  of  the  PBPs.  The  system  extends 
beyond  procedures  and  includes  supporting  displays  for  diagnosing  event  and  plant  status.  The  system  was 
validated  in  tests  involving  ten  crews;  errors  were  reduced  approximately  50  percent  with  the  CBP. 

Mavko  et  al.  (1995)  state  that  classical  paper-based  EOPs  are  not  suitable  for  use  in  CRs  with  digital  process 
information  systems.  They  developed  the  Computerized  Procedure  (COPRO)  system.  COPRO  is  function  oriented 
and  enables  operators  to  restore  and  maintain  critical  safety  functions.  It  includes  the  same  information  as  the 
symptom-based  PBPs,  to  provide  operators  with  necessary  information,  record  their  actions,  and  perform  tasks 


This  type  of  step  is  qualitative  because  its  specification  is  imprecise  in  that  judgement  is  used  to  determine 
the  rate  of  increase  that  is  minimally  necessary  for  operators  to  conclude  the  rate  is  actually  increasing. 
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automatically  -  such  as  monitoring  of  critical  safety  function  (CSF)  status  trees  and  comparing  referenced  and 
actual  values.  The  procedure  continued  automatically  until  stopped.  The  system  was  tested  using  a  small-break 
LOCA  event.  An  operator  completed  the  procedure  more  quickly  with  the  CBP  than  with  the  PBP  and  believed 
that  fewer  errors  would  be  made. 

Observations  of  Operators*  Use  of  CBPs 

The  N4  CBP  system  includes  all  the  N4  procedures,  not  only  the  EOPs.  However,  while  EdF  spent  significant 
effort  on  designing  and  evaluating  the  CBPs,  we  are  not  aware  of  any  papers  with  detailed  results  of  the 
evaluations.  Therefore,  we  reviewed  the  results  of  several  papers  that  discussed  various  lessons  learned  from  the 
evaluation. 

Bozec  et  al.  (1990)  investigated  an  early  version  of  the  N4  CBP  system.  Six  crews  of  operators  participated  in  the 
tests  with  the  N4  simulator.  Their  evaluation  was  mainly  qualitative,  but  from  the  deficiencies  revealed,  they  made 
recommendations  for  improving  the  system.  They  found  that  the  objectives  of  the  procedure  needed  to  be  better 
emphasized  to  increase  the  operators’  awareness  of  the  high-level  goals.  They  suggested  that  providing  too  much 
detail  should  be  avoided,  except  when  there  is  a  problem.  The  operators  did  not  want  the  procedure  to 
automatically  reset  or  return  to  a  previous  step  when  the  status  of  the  process  changed.  However,  they  wanted 
automatic  monitoring  of  previous  steps  and  indications  of  a  change  in  their  status.  The  operator  should  be  able  to 
override  a  course  of  action  that  is  recommended  by  a  CBP  system,  as  when  the  operator  has  access  to  information 
that  is  not  available  to  the  CBP,  the  CBPs  guidance  is  too  strict,  or  the  CBP  is  using  old  information. 

Pirus  and  Chambon  (1997)  offered  additional  lessons  learned  from  EdPs  CBP  evaluations.  Handling  multiple 
procedures  is  easier  when  the  relevant  information  in  each  is  highlighted,  so  that  when  operators  transition  from 
one  to  another,  the  highlighted  information  directs  them  to  the  appropriate  location.  Also,  automatic  monitoring 
of  process  parameters  helps  the  operators.  Finally,  the  quality  of  operations  is  improved  when  operators  2ire  alerted 
to  deviations  from  the  specified  procedure  path,  because  they  then  can  decide  if  that  is  what  they  want  to  do. 

Jeffroy  and  Charron  (1997)  discussed  the  safety  assessment  of  the  EdF  CBP  system  performed  by  France’s  Institute 
for  Nuclear  Safety  and  Protection  (French  acronym  IPSN)  for  the  Nuclear  Installations  Safety  Directorate  (DSIN). 
The  evaluation  was  a  simulator  exercise  which  revealed  several  problems.  While  these  have  been  resolved  in  the 
N4  system,  they  are  important  considerations  for  other  CBP  systems: 

Overall  View  of  the  Process  -  Early  in  the  design,  operators  worked  through  the  flowchart  and  responded  yes  or  no 
to  each  step.  Their  responses  were  monitored  by  a  “path  monitoring  function”  and  deviations  from  the  computer 
were  highlighted.  This  enabled  operators  to  catch  “local”  errors,  but  made  it  easy  to  lose  the  overall  view  because 
of  the  step-by-step  attentional  demands.  By  presenting  procedures  as  a  series  of  pages,  computerization  makes  it 
more  difficult  to  view  the  path  taken,  to  apply  hindsight,  and  to  anticipate  the  consequences  of  an  answer. 

Conflicts  -  While  a  high  degree  of  guidance  can  be  delivered,  not  all  steps  can  be  specified  in  equal  detail.  While 
the  CBP  is  designed  with  steps  in  a  certain  sequence,  operators  sometimes  need  to  alter  it.  Also,  sometimes 
operators  may  disagree  with  the  CBP’s  recommendations.  In  both  situations,  operators  may  find  it  hard  to  disagree 
with  the  procedures,  especially  when  the  level  of  detail  is  high.  CBPs  also  make  it  difficult  to  view  the  path  taken, 
and  this  can  hamper  independence  from  the  procedure.  Operators  would  sometimes  reset  the  procedure  to  get  a 
better  sense  of  how  they  got  to  a  particular  place. 
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Limited  view  -  While  the  CBP  monitors  the  plant  through  instrumentation  and  control  (I&C),  the  operators  must 
supply  some  information;  thus,  the  CBP  may  consider  specific  components  to  be  available  when  they  are  not.  If 
such  communications  fail,  the  CBP  may  make  incorrect  assessments  and  give  incorrect  guidance. 

These  findings  illustrate  the  importance  of  operators  being  aware  of  the  CBP’s  constraints.  However,  some 
situations  made  it  particularly  difficult  to  recognize  them.  For  example,  after  the  crew  negates  a  procedure 
decision,  their  awareness  of  the  basis  for  the  procedure  decisions  becomes  less  clear,  and  the  operators’  and  CBP’s 
“situation  awareness”  begins  to  diverge.  Operators  then  may  not  understand  the  information  provided  nor  the 
effects  of  their  actions  on  the  computer’s  interpretation  of  steps. 

Jefffoy  and  Charron  (1997)  concluded  that  operation  of  procedures  is  a  dynamic  process  involving  interpretation  of 
plant  data,  actions  to  be  carried  out  by  the  plant,  and  interactions  between  crew  members.  Knowledge-based 
understanding  is  needed  to  properly  follow  procedures  and  to  evaluate  the  correctness  of  recommendations. 
Procedure  steps  often  require  the  operators’  input  and  cannot  be  resolved  independently. 

Roth  and  O’Hara  (1998)  studied  the  integration  of  advanced  HSls  into  an  NPP.  During  computer  replacement,  the 
plant’s  CR  was  upgraded  to  include  a  CBP  system,  an  advanced  alarm  system,  and  a  graphic-based  plant  display 
system.  The  authors  observed  crews  during  their  initial  training  with  the  new  systems  on  a  full-scope  simulator, 
and  interviewed  operators  and  other  utility  and  vendor  personnel.  The  training  included  full-scope  simulations  of 
plant  disturbances. 

This  study  was  one  of  the  first  to  evaluate  a  text-based  CBP  system  (previous  studies  were  of  flowchart  CBPs)  and 
one  of  the  first  to  look  at  the  transition  to  PBP  upon  a  CBP  failure.  It  also  was  one  of  the  first  to  examine 
performance  with  a  combination  of  computerized  HSls.  The  results  are  summarized  below. 

(1)  The  general  effect  of  the  CBP  on  performance  was  good.  The  SROs  could  go  through  procedures  more 
quickly,  and  felt  that  their  cognitive  workload  was  reduced  because  information  on  plant  parameters  Wcis 
immediately  available  (the  SROs  did  not  have  to  ask  for  it,  and  operators  did  not  have  to  run  around  to  get 
it),  and  the  SROs  did  not  have  to  resolve  step  logic.  In  general,  procedures  were  followed  more  efficiently 
because  the  operator  was  less  likely  to  miss  a  transition  step  and  did  not  have  to  track  location  within  the 
procedure,  steps  of  continuous  applicability,  applicable  cautions,  or  applicable  foldout  page  criteria.  The 
CBP  was  easy  to  learn,  and  the  operators’  acceptance  was  high. 

(2)  The  CBP  had  an  important  effect  on  the  crew’s  roles  and  communication;  the  extent  of  the  change  was 
greater  than  anticipated  for  board  operators.  Since  the  SRO  could  handle  the  procedure  mainly  alone,  the 
need  for  communication  between  the  SRO  and  ROs  was  reduced.  The  operators  identified  the  importance 
of  communication  in  maintaining  effective  teamwork.  The  ROs  expressed  a  need  to  be  aware  of  status  of 
EOPs.  Because  the  ROs  no  longer  needed  to  support  the  SROs  in  following  the  EOPs  (by  providing 
parameter  values  called  out  in  the  EOPs),  they  had  more  time  and  attentional  resources  available  to 
monitor  the  plant,  giving  them  an  additional  independent  overview  of  its  state.  The  ROs  felt  they  became 
more  independent  and,  thus,  had  more  responsibility.  Therefore,  their  individual  skills  become  more 
important. 

(3)  The  operators’  trust  in  the  CBP  was  high.  They  generally  assumed  that  the  software  logic  was  correct  and 
did  not  feel  a  need  to  double-check  it  by  reading  every  substep.  Instead,  they  double-checked  the  system’s 
conclusions  from  independent  sources  (e.g.,  the  alarms  and  the  board  indicators).  Because  they  generally 
trusted  the  CBP,  they  sometimes  felt  there  was  too  much  information,  preferring  detailed  information  only 
when  a  procedure  step  was  not  satisfied. 
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(4)  Operators  occasionally  wound  up  in  the  wrong  place  when  using  the  CBPs,  such  as  when  step-logic 
resolution  was  oversimplified,  but  usually  recovered  quickly.  Determining  whether  a  step  was  satisfied 
was  sometimes  more  complicated  than  at  first  it  appeared  (e.g.,  interpreting  apparently  simple  statements 
such  as  “If  pressure  is  decreasing...”  can  involve  judgement  that  is  difficult  to  reduce  to  a  simple 
calculation).  Mostly,  crews  could  detect  when  CBP  information  was  inappropriate.  The  operators  were 
generally  tolerant  of  these  situations,  and  felt  that  similar  ones  occur  with  PBPs. 

One  question  was  whether  the  ability  to  identify  such  errors  would  be  the  same  with  crews  that  were  not 
initially  trained  with  PBPs.  Walking  through  paper-based  EOPs  enabled  operators  to  identify  the  goals 
and  logic  behind  them  which  they  could  transfer  in  using  the  CBPs.  CBPs  may  exacerbate  the  tendency  to 
follow  EOPs  verbatim,  without  sufficiently  reflecting  on  the  appropriateness  of  procedure  steps  to  high- 
level  goals. 

(5)  Operators  expressed  initial  concern  over  lowered  situation  awareness  with  CBPs,  but  it  diminished  with 
practice. 

(6)  Operators  did  not  have  a  problem  when  transitioning  to  PBPs  upon  CBP  failure,  although  such  failures 
were  simple  ones  and  happened  early  in  the  EOP. 

The  results  should  be  interpreted  within  the  context  of  the  study’s  constraints:  (1)  the  observations  were  made 
during  the  first  training  period  using  the  new  CBP  systems,  (2)  the  CBP  systems  were  not  completely  debugged, 
and  (3)  the  scenarios  were  limited  to  relatively  simple  events. 

During  an  evaluation  of  CBPs  for  a  low-pressure  injection  system,  Blackman  and  Nelson  (1988)  noted  the 
following: 

(1)  Operators  tended  to  believe  the  computerized  procedure  even  if  it  was  wrong;  they  should  be  trained  to 
question  it. 

(2)  If  selecting  procedures  is  automated,  the  operators’  involvement  was  reduced,  and  they  reported  that  they 
thought  less,  and  acted  as  switch-turners.  Operators  should  be  trained  on  the  decision  process  used  by  the 
procedure  system  and  to  verify  its  recommendations. 

(3)  Operators  continued  attempts  to  implement  a  computer's  recommendation  even  when  failures  prevented  it. 
Operators  need  to  be  trained  to  take  over  if  the  computer  fails  or  is  in  error. 

(4)  Computers  do  not  have  common  sense  functions  employed  by  operators.  The  computer  cannot  consider 
what  operators  are  doing  nor  other  important  information. 

The  major  conclusion  was  that  operators  need  to  understand  the  overall  purpose  of  the  procedures  and  stay 
cognitively  involved  in  their  progress.  They  should  be  trained  to  question  any  recommended  steps  that  appear 
inconsistent  with  the  overall  goals.  Similarly,  CBPs  should  be  designed  to  maintain  information  on  anything  the 
crew  is  doing  that  is  relevant  to  implementing  the  procedure. 

Comparisons  of  CBP  Desien  Characteristics 

In  this  section,  we  discuss  research  on  three  characteristics  of  procedure  design:  presentation  format,  salience 
coding,  and  integration  of  indicators. 
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Presentation  Format 

PBPs  generally  present  procedural  information  in  text  or  as  flowcharts.  A  concern  associated  with  CBPs  is  the 
appropriateness  of  the  presentation  format,  given  the  operator's  task  requirements  and  the  characteristics  of  the 
display  system. 

Wourms  and  Rankin  (1994)  commented  that  text  is  a  sequential  format  that  often  requires  users  to  read 
information  that  is  not  relevant  to  existing  conditions.  Sorting  through  this  information  to  identify  the  correct 
course  of  action  is  time-consuming  and  confusing.  An  important  consideration  in  using  text  is  to  establish  the 
appropriate  level  of  detail.  Operators  rely  more  on  their  memories  than  on  the  actual  procedures  because  the 
narrative  style  uses  too  many  conditional  statements,  which  slows  their  response  time.  Some  systems  overcome 
this  problem  by  providing  information  at  more  than  one  level  of  detail  for  each  step.  For  example,  an  extended 
text  version  of  a  particular  step  may  be  used  by  less  experienced  operators,  while  an  abbreviated  version  may  be 
used  for  familiar  procedures  and  steps. 

In  general,  a  flowchart  format  is  useful  because  of  its  ability  to  specify  the  sequence  of,  and  relationships  between, 
procedures  (Krohn,  1983;  Wourms  and  Rankin,  1994).  Desaulniers,  Gillian,  and  Rudisill  (1988)  compared 
flowchart  formats  to  text  and  extended  text  formats.  Each  was  displayed  in  a  six-line  window.  Peirticipants  were 
asked  to  diagnosis  a  malfunction  in  a  Space  Shuttle  system.  Performance  was  most  accurate  with  the  flowchart 
format.  Overall  completion  times  did  not  differ  between  formats,  but  individual  steps  were  completed  faster  with 
the  flowchart  format. 

In  a  second  experiment,  an  interaction  was  revealed  between  format  and  window  size  (6-  vs.  12-line).  As  the 
window’s  size  increased,  performance  degraded  with  the  flowchart  format,  but  improved  with  the  text  formats.  An 
examination  of  the  errors  that  occurred  suggested  that  participants  lost  their  place  on  the  flowchart  with  the  larger 
display.  For  text  procedures,  the  increased  window  size  helped  users  to  better  understand  the  context  of  procedure 
steps.  The  effect  of  screen  size  upon  accuracy  may  have  important  consequences  when  converting  PBPs  featuring 
flowcharts  to  a  computer-based  medium.  Currently,  paper-based  EOPs  based  on  the  GE  Owners'  Group  technical 
guidelines  have  flowcharts  that  are  the  size  of  engineering  drawings  (Barnes  et  al.,  1996). 

Salience  Coding 

CBPs  are  intended  to  guide  the  operators’  performance  during  plant  upsets  that  may  be  associated  with  time 
pressure  and  stress.  Also,  multiple  procedures  may  be  simultaneously  in  use.  Salience  coding  can  visually 
enhance  presentation  formats,  such  as  text  and  flowcharts.  Color,  flashing,  and  animation  may  be  used  to  enhance 
the  salience  of  important  information.  These  techniques  can  lower  workload  by  helping  to  organize  information 
and  guide  the  operators’  attention  to  that  which  is  most  important. 

However,  because  salience  coding  can  affect  the  operators’  behavior,  care  must  be  given  to  avoiding  coding 
schemes  that  are  distracting,  confusing  or  misleading.  For  example,  Mosier,  Palmer,  and  Degani  (1992)  state: 

The  logical  conclusion  from  the  results  of  research  on  salience  effects  on  decision  making  has  been  that,  in  a 
diagnostic  situation,  the  brightest  flashing  light,  or  the  gauge  that  is  largest  or  most  focally  located  will  bias  the 
operator  toward  processing  its  diagnostic  information  content  over  that  of  other  stimuli.  Time  pressure,  stress, 
or  information  overload  can  cause  a  ^‘perceptual  tunneling”  and  exacerbate  this  tendency  to  focus  on  central  or 
salient  cues.  (p.  10) 
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Integration  of  Indicators 

Incorporating  plant  indications  into  CBPs  poses  both  potential  benefits  and  obstacles  to  human  performance. 

Errors  associated  with  monitoring  the  wrong  display  can  be  avoided  by  providing  the  operator  with  specific 
indications;  Galletti  (1996,  event  4)  describes  the  actuation  of  an  engineered  safety  feature  because  the  operator 
was  monitoring  a  wide-range  instrument  rather  than  a  narrow-range  one.  However,  while  PBPs  force  the  operator 
to  monitor  plant  indications,  incorporating  them  into  CBPs  may  increase  errors  by  becoming  a  substitute  for  good 
monitoring  practices,  or  by  competing  with  other  information  sources  in  the  CR  for  the  operator's  attention.  For 
example,  an  assessment  of  electronic  checklists  (Mosier,  Palmer,  and  Degani,  1992)  concluded  that  those 
encouraging  crews  to  rely  on  the  system’s  state,  as  indicated  by  the  checklist,  rather  than  as  indicated  by  the  system 
itself,  can  discourage  information  gathering,  and  may  lead  to  dangerous  errors.  In  this  aircraft  simulation,  the 
mean  number  of  informational  items  discussed  among  crew  members  decreased  as  the  checklist  became  more 
automated.  Pilots  who  used  paper-based  procedures  were  less  likely  to  shut  down  one  of  the  aircraft's  engines 
unnecessarily. 

Thus,  while  using  CBPs,  operators  may  not  feel  the  need  to  look  at  other  sources  of  information  in  the  CR  and, 
thus,  may  miss  important  indications  that  are  not  present  in  the  CBP  system.  This  need  for  other  information  is 
particularly  important  where  the  system’s  designer  did  not  fully  understand  the  plant’s  behavior,  or  where  the  CBP 
system  fails  in  a  manner  that  is  not  immediately  obvious  to  the  operator.  For  example,  in  events  2  and  3  described 
by  Galletti  (1996),  lockups  of  the  plant’s  alarm  systems  were  only  discovered  after  other  information  sources  were 
compared  to  the  data  provided  by  the  failed  alarm  systems. 

Summary 

In  the  beginning  of  this  section,  we  discussed  criteria  for  assessing  studies.  However,  only  two  studies  were 
described  in  sufficient  detail  to  evaluate  their  generalizability  (Spurgin  et  al.,  1990;  Converse,  1994,  1995).  Both 
had  potential  methodological  weaknesses  which  limit  the  conclusiveness  and  generalizability  of  their  results.  Most 
other  studies  were  not  reported  in  sufficient  detail  to  make  this  evaluation,  or  contained  only  qualitative 
observations. 

Even  considering  the  weaknesses  in  the  design  or  in  the  reporting  of  methodology  and  results,  some  tentative 
conclusions  can  be  drawn  based  on  human  performance  data,  observations,  and  interviews.  On  the  positive  side, 
when  using  CBPs 

•  operators  can  perform  procedures  more  quickly, 

•  operators’  cognitive  workload  seems  to  be  reduced,  and 

•  operators  may  make  fewer  errors  in  transitioning  through  procedures. 

In  addition,  the  CBP  systems  seem  to  be  relatively  easy  to  use  and  are  accepted  by  operators. 

However,  there  remain  several  important,  unresolved  questions  needing  additional  empirical  research: 

•  What  is  the  effect  of  CBPs  on  team  performance  and  reliability? 

•  What  is  the  effect  of  CBPs  on  the  operators’  high-level  situation  awareness  of  the  status  of  the  overall 
procedure  goal  and  the  plant? 
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•  Do  operators  become  over  reliant  on,  and  unquestioning  of  CBPs,  or  can  they  maintain  the  independence  and 
objectivity  to  evaluate  the  adequacy  of  the  procedure  to  achieve  high-level  goals? 

•  What  is  the  overall  effect  on  operators’  errors  of  CBP  systems  (especially  where  errors  are  not  defined  in  terms 
of  verbatim  compliance)? 

•  What  is  the  effect  of  CBPs  on  performance  in  complex  disturbances  involving  many  procedures  or  branches? 

•  How  well  do  operators  manage  complex  CBP  failures,  such  as  when  multiple  procedures  are  being  used,  many 
steps  have  been  completed,  and  many  steps  of  continuous  applicability  are  being  monitored? 

•  What  are  the  relative  effects  of  specific  design  features  on  performance  (most  studies  were  overall  system 
comparisons,  e.g.,  CBP  vs.  PBP,  not  systematic  evaluations  of  individual  characteristics,  such  as  the 
appropriate  level  of  automation)? 

5.4.2  Analytical  Evaluations  of  CBPs 

In  this  section,  two  classes  of  analytical  techniques,  performance  models  and  risk  models,  are  described  that  were 

used  to  evaluate  CBP  design. 

5.4.2.1  Performance-Mode!  Analyses 

CBPs  were  evaluated  using  a  variety  of  performance-analysis  models  including  the  Goals,  Operators,  Methods,  and 

Selection  (GOMS)  model,  MicroSaint  Task  Network  Modeling,  the  Man-Machine  Integrated  Design  and  Analysis 

System  (MIDAS)  model,  and  classical  task  analysis. 

GOMS 


Endestad  and  Meyer  (1993)  compared  COPMA  and  COPMA-11  using  the  GOMS  model  of  HSI  developed  by  Card, 
Moran,  and  Newel  (1983).  As  noted  earlier,  changes  in  the  CBP  included  increased  functionality  for  searching 
procedures;  improvement  in  instructions  for  procedure  steps,  and  more  explicit  references  to  procedure  branches. 
Their  results  indicated  that  the  modifications  resulting  in  COPMA-II  require  additional  learning  and  make  the 
system  more  complex;  however,  COPMA-11  can  be  used  more  rapidly.  They  did  not  identify  the  net  effect  of  this 
tradeoff  on  human  performance. 

MicroSaint  Task  Network  Modeling 

COMPA-11  was  evaluated  using  MicroSaint  Task  Network  Modeling  (Laughery  and  Persensky,  1994) .  The 
operator’s  p>erformance  was  compared  with  experimental  data  collected  on  a  simulator  (Converse,  1995).  The 
comparisons  were  described  as  encouraging;  the  model’s  predictions  of  performance  differences  were  consistent 
with  the  data  in  five  out  of  six  conditions. 

MIDAS 


Hoecker  et  al.  (1994)  and  Hoecker  and  Roth  (1996)  used  the  MIDAS  to  evaluate  CBPs  (Westinghouse’s 
COMPRO)  against  PBPs.  We  note  that  the  primary  objective  of  the  study  was  to  demonstrate  the  application  of 
MIDAS  to  HSI  evaluation.  This  comparison  provided  a  test  case. 
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The  results  indicated  that  the  effect  of  CBPs  on  workload  depends  on  the  situation.  For  example,  the  demands  of 
using  procedures  can  fall  when  the  delays  associated  with  waiting  for  a  response  from  a  board  operator  are 
eliminated.  However,  when  the  operators  need  to  access  information  in  parallel,  the  CBP  system  can  increase  load. 

Task  Analysis 

Niwa,  Hollnagel,  and  Green  (1996)  evaluated  CBP  systems  as  part  of  a  CBP  development  study  for  the  Institute  of 
Nuclear  Safety  Systems  in  Japan.  They  identified  several  reasons  for  their  slow  development.  First,  there  has  been 
a  tendency  to  keep  procedures  separate  from  the  HSI  so  they  constitute  a  “fallback  for  when  all  else  fails.”  Second, 
procedures  are  not  easily  automated  because  they  contain  imprecise  elements  and  depend  on  information  about 
conditions  that  are  not  easily  instrumented. 

Niwa  et  al.  (1996)  made  a  subjective  comparison  of  the  attentional  demands  of  PBPs  and  CBPs  based  on  task 
analysis.  The  basic  tasks  for  using  EOPs  involve  identifying  which  HOP  to  use,  proceeding  step-by-step  through 
the  procedure,  carrying  out  actions  specified  in  steps,  checking-off  completed  steps,  and  retrieving  additional 
information  (from  other  documentation).  The  results  are  summarized  in  Table  5.2  (adapted  from  Table  1  of  Niwa 
et  al.,  1996).  CBP  ratings  were  based  on  a  “well  human-factored  solution”  although  this  was  not  clearly  defined. 

In  general,  they  determined  that  interactions  with  a  CBP  are  more  complicated  than  interactions  with  a  PBP. 

CBPs  may  increase  attentional  demands  in  selecting  the  required  display  (turning  procedure  pages)  and  check¬ 
marking  step  completions.  However,  other  aspects  of  procedure  use  are  easier  with  CBPs,  such  as  retrieving  data. 

Table  5.2  Attentional  Demands  of  PBP  and  CBP  Systems* 


Activity 

PBP 

CBP 

Go  through  steps 

medium 

medium  or  high 

Turn  pages  (select  display) 

very  small 

small  or  medium 

Check-mark  completion 

very  small 

small  or  medium 

Retrieve  additional  information 

very  high 

small  or  medium 

Access  required  source 

medium  or  small 

small 

Find  information 

medium 

small  or  medium 

’Information  is  based  on  the  findings  of  Niwa  et  al.  (1996) 


In  summary,  these  evaluations  show  no  clear  advantage  of  CBPs  over  PBPs.  Instead,  they  illustrate  the  importance 
of  performance  tradeoffs  in  assessing  different  procedure  systems.  In  general,  cognitive  load,  complexity,  and 
attentional  demands  were  higher  for  CBPs,  while  data  retrieval  was  easier  and  task-completion  time  was  less. 

5.4.2.2  Risk-Informed  Analyses 

There  have  been  several  risk-informed  analyses  of  CBPs.  In  one  qualitative  study  of  the  anticipated  impacts  on 
human  performance  introducing  digital  technology  in  NPP  designs,  Wilhelmsen  et  al.  (1992)  identified  several 
concerns.  By  impacts,  the  study  referred  to  potential  changes  in  generic  failure  rates  associated  with  a  crew’s 
performance  in  traditional  systems.  One  of  the  “most  pressing  issues”  identified  was  the  availability  of  on-line 
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procedures.  The  study  also  indicated  that  these  systems  might  improve  performance  if  they  list  procedure  steps, 
logic  flow,  and  allow  simultaneous  access  to  multiple  procedures. 

Two  other  studies  examined  different  aspects  of  performance:  one  evaluated  whether  CBPs  represent  a  potential 
safety-significant  issue  (Stubler  et  al.  1996),  and  the  second  examined  their  potential  effects  on  components  of 
human  error  (Orvis  and  Spurgin,  1996). 

Stubler  et  al.  based  their  safety  evaluation  methodology  on  an  adaptation  of  EPRI’s  approach  in  Guideline  on 
Licensing  Digital  Upgrades  (EPRI,  1993b),  which  was  endorsed  by  the  NRC  in  Generic  Letter  95-02  (NRC,  1995). 
The  following  aspects  of  CBPs  were  associated  with  potentially  negative  effects  on  human  performance: 

•  Level  of  automation  -  The  appropriate  level  of  automation  of  CBP  systems  for  managing  information  is  not 
well  understood. 

•  Design  errors  -  CBPs  that  assess  plant  conditions  and  then  present  corresponding  procedure  steps  may  have 
design  errors  that  stem  from  the  system  designer's  incomplete  understanding  of  the  plzint's  behavior.  These 
errors  may  result  in  inappropriate  analysis  of  information  or  incorrect  guidance  to  the  operator. 

•  Situation  awareness  -  Because  only  a  portion  of  the  procedure  can  be  observed  at  one  time,  operators  may  lose 
a  sense  of  where  they  are  within  the  total  set  of  active  procedures.  The  display  space  may  be  inadequate  to 
allow  simultaneous  viewing  of  multiple  procedures  and  associated  plant  data. 

•  Overreliance  on  CBP  information  -  PBPs  require  operators  to  monitor  plant  indications.  If  these  are  present  in 
the  CBP,  the  operator  may  not  feel  the  need  to  look  at  other  sources  of  information  in  the  CR  and,  thus,  may 
miss  important  indications  that  are  lacking  in  the  CBP. 

•  Navigation  -  Navigation  within  one  procedure,  or  among  multiple  ones  and  related  supporting  information, 
can  be  time  consuming  and  error  prone. 

•  Computer-based  text  characteristics  -  In  general,  comparisons  of  task  performance  for  information  presented 
via  VDUs  and  hard  copy  indicate  that  there  are  significant  differences  between  them.  Reading  is  generally 
slower  and  more  fatiguing  using  VDUs,  and  they  have  been  associated  with  poorer  performance  and  lower 
usability  ratings. 

•  Salience  coding  -  Presentation  formats,  such  as  text,  flowcharts,  and  hypertext,  can  be  visually  enhanced  by 
the  graphical  capabilities  of  computer-based  displays.  For  example,  color,  flashing,  and  animation  can 
enhance  the  salience  of  important  information.  However,  improper  coding  can  have  negative  effects  on 
operators’  behavior  by  de-emphasizing  or  drawing  attention  away  from  important  information.  Thus,  the 
design  of  coding  schemes  is  critical  for  successfully  implementing  CBPs. 

•  Consistency  with  the  HSl  -  Any  inconsistency  of  the  CBP  with  the  rest  of  the  HSI  can  lower  performance  and 
increase  the  likelihood  of  errors.  Some  important  aspects  include  the  degree  to  which  the  display  of  plant 
variables  and  units  of  measurement  used  in  the  CBP  are  the  same  as  in  the  normal  monitoring  displays,  the 
same  coding  schemes  are  used,  and  navigation  mechanisms  are  compatible  with  those  of  other  display  devices 
in  the  HSI. 

•  Transfer  between  CBPs  and  PBPs  -  Under  some  circumstances,  the  operators  may  be  required  to  switch 
between  PBPs  and  CBPs.  For  example,  if  the  normal  operating  procedures  are  given  in  paper-based  form  and 
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the  EOPs  in  computer-based  form,  then  the  operator  must  switch  to  CBPs  when  the  EOPs  are  to  be  used.  If 
the  CBP  system  fails,  then  the  operators  may  be  required  to  use  PBPs.  Transfers  between  them  may  be 
difficult,  especially  if  their  formats  (e.g.,  flowchart  versus  text  format)  and  mechanisms  for  managing  the  use 
of  multiple  procedures  (e.g.,  “place  holders”  and  navigation  features)  differ.  The  operator’s  burden  is  likely  to 
increase  with  such  switching  when  other  demands  are  high  (e.g.,  as  a  result  of  the  condition  that  required 
using  EOPs). 

Orvis  and  Spurgin  (1996)  evaluated  CBPs  from  a  perspective  of  a  cognitive  reliability  model.  We  note  the  analysis 
assumed  that  CBPs  had  positive  effects  on  performance  and,  therefore,  it  was  aimed  at  where  improvements  in 
crew  reliability  can  be  expected.  For  example,  Moieni  and  Spurgin  (1993b)  have  noted  that 

...computers  can  make  up  for  some  of  the  human  limitations,  such  as  short  term  memory  and  limited 
working  memory  capacity,  and  together  with  the  human  operator  can  be  more  effective  and  reliable 
than  either  acting  separately.  Thus,  computers  can  help  the  user  find  his  way  through  the  procedures 
and  help  ensure  that  steps  in  the  procedures  are  taken  in  the  correct  sequence.  More  importantly, 
they  can  support  the  crew  in  taking  into  account  the  correct  set  of  symptoms,  and  help  ensure  that 
key  elements  are  not  ignored.  In  some  systems,  the  computer  can  take  control  if  the  crew  fails  to 
follow  the  procedures  as  prescribed. 

The  cognitive  model  had  two  separate  phases:  the  detection-diagnosis-decision  making  (DDD)  phase,  and  the 
implementation  phase.  There  are  three  pathways  to  failure  to  provide  the  correct  response  within  the  required 
time.  First,  the  crew  may  fail  to  detect  the  need  to  take  action,  or  may  make  a  misdiagnosis  (P,).  Second  is  the 
failure  to  complete  the  DDD  phase  within  the  required  time  The  third  path  is  the  crew’s  failure  to  complete 
all  required  actions  (P3).  The  total  probability  of  human  error  for  a  given  human  interaction  is 

P  Humjo!  ”Pi  +  p2"^P3 

Orvis  and  Spurgin  felt  that  the  CBP  should  reduce  the  probability  of  all  failure  pathways.  Since  the  CBP 
automatically  detects  parameters  and  matches  them  to  EOP  conditions,  Pj  could  be  essentially  eliminated.  When 
CBPs  monitor  whether  an  action  was  taken  and  notify  the  crew  if  it  is  still  needed,  P3  can  be  essentially  eliminated 
as  well.  Thus, 


PHun..Tot(CBP)  <  Phu..To.(PBP) 

Orvis  and  Spurgin  (1996)  have  determined  that  the  following  features  of  CBPs  will  affect  crew  reliability: 

•  Quality  of  display 

•  Number  of  windows  concurrently  open 

•  Coupling  with  plant  parameters 

•  Coupling  with  alarms 

•  Display  of  control  status 

•  Display  of  plant  mimics  with  component  status 
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•  Automatic  EOF  selection 

•  Easy  navigation 

•  Similarity  of  operation  for  normal  and  abnormal  procedure  use 

•  Automatic  place  keeping  in  EOPs 

•  Limited  amount  of  user  configuration 

•  No  lockup  on  erroneous  use 

The  analysis  by  Orvis  and  Spurgin  was  to  be  an  assessment  of  the  potential  benefits  of  CBPs.  Potentially  negative 
factors,  such  as  those  examined  by  Stubler,  Higgins,  and  O'Hara  (1996),  were  not  examined.  They  are  careful  to 
point  out  that  improvements  in  reliability  have  to  be  made  using  the  results  from  simulations. 

While  the  evaluations  of  performance  models  showed  no  clear  advantage  of  CBPs  over  PBPs,  the  risk-oriented 
analyses  show  that  while  CBPs  have  the  potential  to  increase  the  reliability  of  human  performance,  when  poorly 
implemented,  they  can  reduce  it, 

5.43  Expert  Opinion 

This  section  reviews  literature  that  discusses  CBPs  where  the  findings  are  based  on  the  opinion  of  subject-matter 
experts  (SMEs),  rather  than  specific  data  collection  or  analyses.  It  includes  the  NRC-sponsored  review  of  CBP 
systems  by  Spurgin  et  al.,  the  NRC  CBP  workshop,  and  an  IAEA  working  group  on  computerization  of  CRs  that 
covered  CBPs. 

NRC-Sponsored  Review  of  CBP  Systems 

Spurgin,  Wachtel,  and  Moieni  (1993)  reviewed  several  CBP  systems  based  on  a  literature  review,  a  questionnaire, 
and  interviews.  Their  findings  indicated  that  CBPs  have  important  impacts  on  NPP  operations,  some  of  which 
extend  beyond  those  intended  by  the  designers.  The  change  from  PBPs  to  CBPs  may  affect  the  crew’s  structure, 
human  reliability,  training,  and  selection  criteria.  They  identified  the  following  general  findings: 

•  CBPs  can  perform  many  tasks  typically  undertaken  by  multiple  crew  members;  these  include  monitoring 
functions,  selecting  a  procedure,  selecting  procedure  steps  based  on  the  plant’s  state,  and  providing  the 
rationale  for  the  choices.  Thus,  CBP  use  will  require  a  single  crew  member. 

•  EOPs  are  used  differently  in  different  countries;  therefore,  their  design  will  reflect  these  differences. 

•  Recovery  from  human  error  is  faster  with  CBPs  than  with  PBPs. 

•  CBPs  appear  to  be  beneficial  during  multiple  failures. 

•  CBPs  may  introduce  new  types  of  errors  related  to  software  errors  and  those  due  to  the  designer’s  incomplete 
understanding  of  the  plant. 

•  Thorough  V&V  is  a  crucial  requirement. 
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•  The  extent  of  system  automation  and  allocation  of  functions  between  the  CBP  and  crew  are  impxjrtant,  but  may 
not  be  sufficiently  considered  by  designers. 

•  CBPs  will  significantly  affect  the  administrative  control  and  configuration  control  of  plant  procedures. 

•  The  way  in  which  CBPs  are  introduced  into  plants  and  the  training  that  operators  receive  are  very  important. 
For  example,  it  is  unknown  whether  CBPs  should  be  introduced  into  training  simulators  first,  or  whether  they 
should  be  given  to  novice  crews  only. 

It  was  concluded  that  “...  more  work  needs  to  be  done  before  the  industry  can  make  a  safe  transition  from 
traditional  paper  and  pencil  procedures  to  computerized  systems”  (p.  1017). 

Spurgin  (1995)  discussed  the  effect  of  computerizing  EOPs  on  the  operator’s  role.  Two  classes  of  CBPs  were 
identified.  The  first  presents  EOP  information  to  the  operators,  and  they  decide  on  the  subsequent  actions.  The 
second  class  recommends  how  to  proceed,  and  the  operator  confirms.  The  latter  minimizes  the  operator’s  role  and, 
according  to  Spurgin,  may  be  a  deterrent  to  taking  action. 

NRC  CBP  Workshop^ 

The  NRC  conducted  a  major  workshop  on  CBPs  in  1 994  to  identify  the  key  issues  that  need  to  be  resolved  to 
support  HFE  guidance  for  reviewing  CBPs.  Fifteen  participants  were  identified  as  SMEs  by  an  international 
selection  process.  Individual  presentations  on  the  current  status  of  CBP  systems  development  and  resezirch  were 
discussed.  The  SMEs  were  divided  into  two  working  groups  to  identify  the  issues.  The  results  are  summarized 
below. 

CBP  Taxonomy 

A  taxonomy  to  describe  CBP  features  and  functions  is  needed  to  support  regulatory  evaluations  and  regulators  in 
exploring  differences  in  systems. 

Automation  and  Task  Allocation 

While  it  is  important  for  operators  to  be  in  control  of  the  CBP,  they  may  become  more  complacent  or  dependent  on 
the  CBP  and  fail  to  consider  whether  it  is  malfunctioning.  The  extent  of  the  crew’s  monitoring  should  be 
considered.  The  CBP  should  inform  operators  of  disagreements,  but  should  allow  them  to  take  actions  if  they  want 
to  override.  The  crew’s  actions  should  be  logged. 

The  degree  to  which  specific  CBP  features  should  by  automated  is  unknown  and  should  be  assessed. 

Crew  Performance  and  Coordination 

The  impact  of  CBPs  on  task  performance  of  crew  members  needs  careful  assessment;  CBPs  should  support 
cooperation,  interaction,  and  decision  making.  Beyond  these  generalities,  several  specific  issues  were  identified: 


^The  findings  from  the  NRC  CBP  Workshop  are  being  documented  in  a  report  that  is  currently  in  draft  form  (Kancler, 
D.,  Schopper,  A.,  and  Wachtel,  J.  Findings  of  a  workshop  on  computer-based  procedures  in  nuclear  power  plant  control 
rooms.  Washington,  D.C.:  U.S.  Nuclear  Regulatory  Commission.)  For  the  purposes  of  our  research,  any  findings  from  the 
workshop  that  contributed  to  CBP  design  review  guidance  development  are  included  in  the  summary  provided  in  this  section. 
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•  An  operator’s  situation  awareness  should  not  be  adversely  affected  by  CBPs.  Two  aspects  2ire  important: 
awareness  of  the  plant’s  status  (information  should  be  available,  and  interacting  with  CBPs  should  not 
interfere  with  situation  assessment),  and  procedural  aweireness  (aweireness  of  procedure  goals,  how  they  sure 
structured,  and  knowledge  of  the  location  within  a  procedure  or  between  a  set).  Situation  awareness  should  be 
measured  in  evaluating  the  systems. 

•  High  workload  may  be  a  concern. 

•  Use  of  CBPs  should  be  consistent  with  normal,  daily  operations  to  be  effective. 

•  CBPs  should  support  cross-checking. 

Training 

The  SMEs  noted  that  training  was  critical  to  the  success  of  CBPs,  but  that  significant  changes  may  be  required. 
VDUs  may  introduce  effects  such  as  glare  and  eyestrain.  They  also  offer  features  not  found  in  PBPs.  Training 
should  address  the  procedures’  structure,  conventions,  and  rules  of  use. 

Training  also  should  focus  on  limitations  of  the  system  and  establishing  the  operators’  trust.  However,  operators 
should  be  trained  to  minimize  overdependence  or  reliance  on  CBPs,  and  also  on  the  proper  means  of  handling 
disagreements  between  the  crew  and  the  CBP  system  about  appropriate  actions.  In  addition,  operators  need  to  be 
trained  to  detect  failures  and  on  both  CBPs  and  their  backup  systems.  The  evaluation  of  training  should  ensure 
that  any  effects  of  negative  transfer  are  minimized. 

Human-System  Interface 

Several  aspects  of  HSI  design  were  discussed.  CBPs  should  represent  procedure  attributes,  such  as  steps  of 
continuous  applicability  and  transitions,  and  should  provide  navigational  means  to  access  different  parts  of  the 
procedure,  different  procedures,  and  additional  information.  Another  issue  identified  was  the  appropriate 
application  of  the  computer’s  capabilities  (color  coding,  animated  graphics,  and  video)  to  procedures. 

Desirable  characteristics  of  CBPs  included  the  capability  to  adjust  the  level  of  detail  and  to  annotate  the  CBP.  The 
degree  of  flexibility  in  the  CBP  HSI  was  identified  as  a  concern;  such  flexibility  should  not  affect  procedure 
information.  Consistency  with  other  HSIs  also  was  identified  as  an  important  requirement. 

An  issue  was  raised  as  to  whether  the  computer  medium  may  affect  how  operators  interpret  procedures.  For 
example,  some  procedural  details  intentionally  are  left  abstract.  While  computerization  may  allow  an  increase  in 
detail,  the  result  could  be  an  inadvertent  change  in  the  procedure’s  context  and  its  interpretation. 

CBPs  should  be  able  to  support  improvements  in  procedures.  Gaps  occur  in  procedures  because  they  do  not  cover 
all  possible  situations  and  actions;  CBPs  allow  operators  to  log  their  occurrence  in  an  on-line  database.  The 
database  could  be  accessed  to  identify  aspects  of  the  procedure  that  need  improvement. 

System  Reliability 

Failure  modes  (bugs,  logic  failures,  and  bad  input)  need  careful  evaluation  along  with  how  operators  can  detect 
those  that  were  not  corrected  during  design. 
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Verification  and  Validation 

CBP  V&V  evaluations  should  provide  evidence  that  operators  can  perform  their  tasks  in  real  time.  They  should 
involve  procedure  guidance  (such  as  NUREG-0899),  usability  testing,  findings  of  prior  research,  and  realistic 
scenarios.  V&V  should  address  CBP  failure,  the  transition  between  PBPs  and  CBPs,  the  introduction  of  CBPs  into 
a  PBP  CR,  and  using  PBPs  if  the  CBP  fails.  Finally,  V&V  for  software  should  be  clarified. 

IAEA  Working  Group  on  CR  Computerization 

Similar  to  Spurgin  (1995),  the  IAEA  working  group  on  CR  computerization  concluded  that  CBPs  potentially  can 
minimize  the  operator’s  role  but  may  deter  people  from  taking  action  (IAEA,  1995).  They  considered  CBPs  to  be  a 
future  trend.  Some  of  the  advantages  noted  were  that  (1)  information  will  be  integrated,  (2)  events  will  be 
confirmed  (e.g.,  CBP  can  indicate  if  a  procedure  step  is  satisfied),  and  (3)  information  will  be  context  sensitive 
(procedures  can  inform  operators  based  on  the  current  state,  e.g.,  they  will  not  display  the  step,  “turn  on  Pump  A,” 
if  Pump  A  is  on  already).  CBPs  should  guide  systematic,  rapid  implementation  of  procedures.  However,  the  IAEA 
believed  that  current  usage  of  CBPs  is  “minimal  and  in  its  infancy”  and,  therefore,  recommended  that  CBPs 
“should  be  developed  as  research  projects  and  prototypes,  and  feasibility  tested  on  suitable  full-scale  application 
where  these  may  be  possible”  (p.  62). 

Summary 


The  SME  examination  of  CBPs  identified  many  positive  aspects  of  CBPs’  use  on  crew  performance.  However, 
SMEs  also  identified  a  wide  range  of  unresolved  issues  which  partly  have  led  to  the  conclusion  that  CBPs  should 
be  introduced  carefully  into  operational  plants.  The  issues  identified  should  be  considered  in  developing  CBPs.  A 
noteworthy  observation  is  that  CBP  development  must  consider  related  HFE  activities,  such  as  training,  and 
integrating  the  CBP  system  with  the  other  HSIs  and  the  overall  operational  philosophy  of  the  plant.  V&V 
programs  are  again  emphasized.  In  general,  these  findings  are  fully  consistent  with  the  other  sources  of 
information  discussed  earlier. 

5.5  Other  Related  Research  on  Computerization  of  Task  Aids 

Two  areas  of  research  will  be  discussed  that  provide  insights  generalizable  to  procedures,  while  not  specifically 
addressing  their  computerization.  The  first  research  includes  general  comparisons  of  task  performance  using 
computerized  versions  of  what  historically  were  hard-copy  support  materials.  The  second  area  is  the  topic  of 
computerized  operator  support  systems  (COSSs). 

5.5.1  General  Comparisons  of  Hard-Copy  and  Computer-Based  Task  Performance 

In  this  section,  we  briefly  review  the  general  research  literature  comparing  hard-copy  and  computer-based 
presentations  of  the  same  information.  While  there  have  been  many  such  comparisons  for  reading,  there  have  been 
few  studies  of  the  effects  on  Job  performance  (of  which  reading  is  only  a  component),  such  as  maintaining 
equipment. 

Reading  is  an  important  part  of  any  task  in  which  information  is  presented  to  users  on  a  computer,  and  an 
important  aspect  of  procedure  use.  Therefore,  a  great  deal  of  research  was  devoted  to  comparing  reading 
performance  on  computers  vs.  typical  hard  copies.  Generally,  reading  is  slower  and  more  fatiguing  with  VDUs 
than  with  a  hard  copy  of  the  same  material  (Gould  et  al.,1987). 
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Examining  the  computerization  of  technical  manuals,  Shneideiman  (1987)  identified  several  potential 
disadvantages: 

•  Computer  screens  are  not  as  readable  as  printed  material. 

•  Computer  screens  provide  less  information  than  paper,  and  also  the  rate  of  paging  is  slower. 

•  The  need  to  use  a  computer  interaction  technique,  such  as  command  language,  and  navigation,  requires  more 
mental  effort  and  may  interfere  with  the  primary  tasks. 

•  If  the  display  screen  is  used  for  other  work,  users  may  have  to  switch  back  and  forth  between  the  computerized 
manual  and  other  information. 

To  examine  these  potential  issues,  Shneideiman  reviewed  several  studies  of  computerized  manuals;  he  concluded 
that  performance  may  not  improve  and  may  actually  degrade.  Thus,  Shneideiman  concluded  that  “At  this  stage  of 
technology,  paper  manuals  are  still  preferred”  (p.  382).  However,  this  conclusion  was  based  on  1980's  data. 

We  reviewed  several  more  recent  studies  which  compared  task  performance  using  computer-based  and  hard-copy 
aids.  Consistent  with  Shneiderman’s  conclusions,  they  generally  found  that  computer-based  presentations  are 
associated  with  slower,  poorer  task  performance  (e.g.,  Reaux  and  Williges,  1988;  Fox,  1992)  and  the  use  of 
different  task  strategies  (e.g.,  Ogawa  and  Yonemura,  1992). 

Nelson  and  Smith  (1990)  set  up  repair  manuals,  including  text  and  graphics  for  mining  equipment,  in  HyperCard 
on  an  Apple  Macintosh  computer.  Subjects  performed  tasks  using  either  the  computer-based  or  the  hard-copy 
manuals.  The  first  task  required  subjects  to  complete  written  statements  by  searching  the  manual  and  filling  in  the 
exact  information  that  was  found  in  it.  The  second  task  required  subjects  to  answer  multiple  choice  or  true-false 
questions  on  nine  realistic  maintenance  situations.  They  then  were  asked  about  their  personal  preferences  to  assess 
their  acceptance  of  the  modes  of  information  presentation,  and  how  it  compared  to  other  manuals  they  had  used. 
Subjects  using  the  computer-based  manuals  were  significantly  slower  and  finished  fewer  of  the  tasks,  but 
performed  much  more  accurately  on  the  parts  they  completed.  While  those  using  the  on-line  manual  considered  it 
harder  to  use,  subjective  evaluations  were  positive  (“quick  response,  good  illustrations,  compact,  fiin  to  use,  finding 
general  subjects  area  and  Word  Find  are  very  helpful,  and  no  greasy,  dirty,*  tom  pages”).  The  problems  identified 
included  annoyance  due  to  its  brightness,  eyestrain,  and  headaches. 

Federico  (1991)  tested  Navy  subjects’  identification  of  Soviet  and  U.S.  planes  using  either  computer-based  or  hard¬ 
copy  presentations.  The  two  modes  did  not  differ  in  accuracy  or  internal  consistency;  however,  the  subjects’ 
confidence  in  their  recognition  was  greater  when  using  the  hard-copy  presentation.  As  the  experimenters 
hypothesized,  “...the  longer  exposures  intrinsic  to  the  paper-based  method  seemed  to  have  facilitated  subjects’ 
recognition  scores.  They  performed  significantly  better  on  the  paper-based  test  than  the  computer-based  test.” 

Krauss,  Middendorf,  and  Willits  (1991)  compared  one  group  of  subjects  who  learned  to  use  a  software  product 
through  an  on-line  tutorial  with  another  group  learning  the  same  tasks  using  a  hard-copy  tutorial.  Subjects  were 
given  a  sample  application  task  and  a  main  application  task.  In  the  first,  they  were  led  explicitly  through  the 
actions  necessary  to  complete  the  task  in  a  cookbook  fashion.  Immediately  afterwards,  they  began  the  main  task, 
which  required  them  to  develop  an  application  that  allowed  entry  of  information  about  employees  (such  as  names, 
social  security  numbers,  and  job  classifications).  To  accomplish  this  task,  they  had  to  specify  tables,  define 
records,  and  create  screens,  and  were  expected  to  refer  back  to  their  respective  tutorials.  Subjects  working  on-line 
were  slower  and  found  it  more  difficult  than  did  those  with  the  hard  copy.  This  was  due  to  navigational  problems 
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associated  with  their  confusion  with  manipulating  windows  and  finding  information  on  hidden  screens;  they 
reported  a  “lost”  feeling.  The  authors  hypothesized  that  providing  an  outline  of  the  entire  document  (e.g.,  in  the 
comer  of  the  screen)  and  highlighting  the  user's  location  in  it  might  mitigate  this  problem. 

Weldon,  Koved,  and  Shneiderman  (1985)  compared  two  types  of  information  structure:  linear  (usually  found  in 
books)  and  tree  (browsing  through  specific  titles  and  finding  details  elsewhere).  Subjects  read  from  four  different 
versions  (one  for  each  of  the  experimental  conditions:  online-linear,  online-tree,  hzird  copy-linear,  and  hard  copy- 
tree)  of  a  simulated  electronic  intercom-maintenance  manual,  written  for  the  experiment.  Each  version  was 
identical  in  content,  but  organized  differently.  The  subjects  were  asked  to  determine  the  correct  settings  for  two 
sets  of  eight  dip-switches  soldered  to  a  prototyping  card.  The  problems  required  different  combinations  of  on  and 
off  switch  settings.  The  dependent  variables  were  the  time  to  solve  the  problems,  the  number  of  errors,  the  number 
of  pages  viewed,  and  the  subjective  evaluations.  It  was  found  that  the  information’s  structure  did  not  affect 
performance.  Instead,  the  important  variable  was  whether  the  subject  had  read  from  the  on-line  manual  or  from 
the  hard-copy  manual;  subjects  using  hard  copies  were  faster.  Within  the  online  condition  itself,  there  was  a 
significant  difference  in  the  number  of  pages  viewed;  subjects  given  tree-structured  information  looked  at  more 
pages  than  people  given  linear  information.  There  were  no  significant  differences  in  the  number  of  errors  in 
switch-setting  combinations  among  the  experimental  conditions.  In  the  subjective  evaluations,  subjects  preferred 
the  on-line  mode  over  the  paper  mode,  but  there  were  no  significant  differences  in  type  of  information  structure 
preferred.  The  experimenters  hypothesized  that  structure  may  be  more  important  in  studies  which  used  larger 
manuals,  but  this  suggestion  was  not  tested. 

Kincaid,  Schurman,  and  Hays  (1990)  compared  a  paper  maintenance  manual  with  a  computer-based  manual,  the 
Portable  Electronic  Aid  for  Maintenance  (REAM),  observing  technicians  in  a  tank-maintenance  task.  The  results 
indicated  that  use  of  the  electronic  system  resulted  in  only  about  1/3  of  the  errors  of  the  paper  manuals;  however, 
the  time  to  perform  the  task  was  slightly  longer,  which  was  attributed  to  computer  delays  in  presenting 
information.  Based  on  the  PEAM  results,  Inaba  (1990)  identified  several  lessons  learned: 

•  HFE  principles  for  paper  presentations  also  apply  to  electronic  presentations. 

•  The  major  advantage  of  the  electronic  manual  was  its  ability  to  display,  store,  and  retrieve  large  amounts  of 
data. 

•  The  major  hardware  limitation  of  a  restricted  screen  area  can  be  overcome  by  applying  HFE  principles. 

One  study  found  a  positive  effect  of  computerization.  Andre  and  Pouraghabagher  (1995)  compared  computer 
checklists  to  paper  checklists  for  missile  Launch  Control  Center  tasks.  The  computer-based  formats  reduced  the 
response  time  of  expert  operators  by  10  percent,  and  substantially  reduced  their  error  rates  (by  58  percent) 
compared  to  the  paper-based  system.  The  effects  were  not  as  marked  with  non-experts. 

Summary 

It  can  be  concluded  from  the  general  literature  that  task  performance  differs  when  information  is  computer  based 
versus  when  it  is  presented  as  hard  copy.  Reading  from  a  VDU  is  generally  slower  and  more  fatiguing.  VDU- 
based  complex  task  performance  also  was  associated  with  poorer  performance  and  problems  in  usability. 
Contributing  to  these  differences  are  difficulties  in  maintaining  a  sense  of  location  (knowing  where  you  are  in  a 
document),  navigation  (moving  from  one  place  in  a  document  to  another),  and  fatigue.  Chignell  and  Zuberec 
(1993)  noted  similar  potential  difficulties  with  use  of  CBPs:  visual  fatigue,  glare,  and  resolution. 
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5.5.2  Computerized  Operator  Support  Systems  (COSSs) 

Numerous  COSSs  based  upon  knowledge-based  systems  (KBSs),  such  as  expert  systems,  assist  in  cognitive  tasks 
such  as  evaluating  plant  conditions,  diagnosing  faults,  and  selecting  response  strategies.  Intelligent  aids  may 
include  (1)  automatic  checks  which  track  operators’  actions  and  compare  them  to  actions  expected  from  plant 
procedures  or  other  models,  (2)  automatic  warnings  based  on  current  conditions,  predicted  consequences  or  side 
effects,  and  (3)  smart  interlocks  that  block  control  actions  that  conflict  with  the  plant’s  current  configuration. 

The  nuclear  industry  has  developed  a  wide  range  of  KBS  applications  for  off-line  analysis  and  on-line  cognitive 
support  to  plant  personnel  (IAEA,  1993,  1995): 

•  Fault  detection  and  diagnosis 

•  Safety  function  monitoring  (e.g.,  severity  of  challenges  to  critical  safety  functions) 

•  Plant-performance  monitoring  (e.g.,  efficiency  of  main  pumps,  turbine,  and  generator) 

•  Core  monitoring 

•  Advising  on  unforeseen  maintenance  problems 

•  Interpretation  of  complex  procedures  or  regulations 

•  Support  for  controlling  the  plant 

Several  off-line  systems  have  been  applied  to  areas  related  to  safety.  The  overall  trend  in  the  nuclear  power 
industry  appears  to  be  a  move  from  conventional  off-line  applications  toward  on-line  systems.  The  principal  area 
of  application  appears  to  be  fault  diagnosis,  which  requires  a  monitoring  capability.  A  variety  of  computer-based 
aids  that  analyze  plant  conditions  and  then  make  recommendations  to  personnel  (e.g.,  for  improving  plant 
performance,  diagnosing  failures,  and  identifying  success  paths)  are  discussed  in  the  literature;  many  of  them  are 
research  prototype  systems.  For  example,  expert  systems,  based  on  artificial  neural  netwoiic  technology,  were 
developed  for  the  following  NPP  applications:  diagnosing  faults,  analyzing  core  vibrations,  monitoring  loose  parts, 
modeling  thermodynamics,  estimating  thermal  margins,  and  identifying  transients  (Uhrig,  1994).  The  commercial 
uses  of  computer-based  aids  include: 

•  Emergency  response  projection  code  -  Software  for  projecting  doses  that  would  be  received  by  the  areas 
surrounding  a  nuclear  generating  station  in  an  accident  involving  airborne  release  of  radioactive  materials. 
[The  Pickering  Emergency  Response  Projection  computer  code  is  described  in  AECB  (1994).] 

•  Fuel  loading  expert  system  -  A  computer-assisted  system  for  fuel  reloading  while  at  power  was  designed  for 
CANDU  NPPs  (Gertman  et  al.,  1994). 

AECL  has  several  aids  under  development  (O’Hara,  Stubler,  and  Higgins,  1996): 

•  Advanced  Process  and  Analysis  Control  System  -  This  is  a  rule-based  computer  system  that  assists  op>erators 
and  maintenance  personnel  with  on-line  diagnosis  of  process  and  equipment  faults.  The  prototype  system  was 
applied  to  the  CANDU  Bruce  B  feedwater  system. 
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•  Feedwater  Corrosion-Monitoring  and  Prediction  Analysis  System  -  This  system  supports  the  detection, 
monitoring,  diagnosis,  and  prediction  of  corrosion  problems  in  the  secondary  side  of  a  CANDU  plant,  based 
on  chemical  analyses.  Neural  nets  are  used  in  the  diagnosis  portion  of  the  system. 

•  A  signal-analysis  system  for  calibrating  trip  channel  signals. 

•  A  virtual-reality -based  system  for  visualizing  the  interior  of  a  reactor  fuel  channel  to  support  the  removal  of 
stuck  fuel  bundles.  This  system  is  envisioned  as  a  training  aid. 

AECB  described  an  operator  decision  aid  currently  under  development  in  Canada  that  simulates  plant  performance 
using  an  ideal  model.  It  continuously  compares  actual  plant  values  to  simulated  values  to  identify  plant  systems 
that  may  be  degrading  or  failing. 

Japan  Atomic  Power  Company's  Tsurugan  NPP  (Unit  2)  includes  a  Mitsubishi  Computerized  Operator  Support 
System  (MCOSS).  Its  objective  is  to  aid  operator’s  decision  making  by  detecting  abnormal  operating  conditions 
before  they  become  serious  and  to  advise  the  operator  of  appropriate  actions.  If  its  early  warning  capability  does 
not  prevent  a  plant  trip,  the  MCOSS  assists  the  operator  in  reaching  safe  shutdown.  The  utility  was  concerned  that 
measurement  noise  would  impair  the  system’s  diagnostic  capability;  however,  this  was  not  the  case.  The  system’s 
response  time  is  approximately  6  to  7  seconds,  which  was  judged  acceptable,  although  shorter  times  were 
preferred. 

This  system  will  be  further  developed  in  the  Mitsubishi  Advanced  PWR.  Mitsubishi  is  developing  a  KBS  for  use 
during  accident  conditions  and  when  operators  are  under  high  stress.  The  system  develops  hypotheses  about  plant 
conditions  based  upon  the  available  symptoms,  and  then  tests  each  one.  It  uses  a  windows  interface  with  dialogue 
control  at  the  bottom  of  the  screen.  The  operator  can  request  an  explanation  of  the  system’s  hypothesis,  procedural 
guidance,  and  evaluation  of  alternative  solutions  before  actions  are  taken.  It  is  anticipated  that  use  of  the  system  in 
a  real  CR  would  be  a  full-time  job,  probably  for  a  senior  operator. 

Despite  the  development  of  many  COSSs,  there  is  not  much  experience  with  operational  aspects  of  their  use. 
Several  experimental  evaluations  of  the  value  of  expert  systems  to  operators  were  inconclusive  (Bernard  and 
Washio,  1989).  Furthermore,  there  is  a  trend  for  expert  systems  to  be  abandoned  after  prototype  testing  and  brief, 
in-plant  trials.  The  transition  from  a  prototype  system  to  a  production-grade  product  requires  a  significantly 
greater  effort  than  initial  prototyping  (Cain  and  David,  1 989). 

As  discussed,  the  predominant  role  of  COSSs  has  been  as  decision  aids.  In  this  role,  perhaps  the  most  significant 
factors  are  intelligibility  and  communication  (IAEA,  1995;  Malin  et  al.,  1991a,  1991b;  Land  et  al.,  1995;  Rook  and 
Donnell,  1993).  It  is  essential  to  the  operators’  acceptance  and  use  of  COSSs  that  the  reasoning  process  is  fully 
understood  by  those  using  them.  Personnel  must  be  able  to  communicate  both  ways  with  the  KBS  COSSs,  i.e.,  the 
degree  to  which  the  bases  for  its  results  are  given  and  to  which  operators  can  query  the  system  when  its  results  are 
not  understood  is  important. 

Rook  and  Donnell  (1993)  experimentally  manipulated  the  interface  of  an  expert  system  which  was  designed  to 
support  fault  diagnosis  during  simulated  space  station  problem-solving  situations  to  make  its  reasoning  more  or 
less  intelligible  to  the  subjects.  They  found  that  users  of  the  system  had  to  have  a  good  mental  model  of  its 
reasoning  process  to  use  it  effectively.  Since  understanding  the  system  was  critical  to  its  use,  they  predicted  that 
the  design  of  the  display  would  be  a  significant  factor  in  using  the  system. 
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Malin  et  al.  (1991a)  discussed  case  studies  of  the  design  of  1 5  intelligent  systems  developed  for  a  variety  of 
aerospace  projects.  The  systems  mainly  were  real-time  fault  management  systems.  The  interfaces  between  the 
human  operator  and  intelligent  systems  were  found  to  be  problematic.  Some  specific  concerns  identified  were 
quite  similar  to  those  identified  above:  providing  visibility  into  the  system’s  reasoning,  understanding  its 
reasoning,  the  system’s  response  in  the  context  in  which  a  question  is  asked,  distinguishing  hypotheses  from  facts, 
determining  the  credibility  and  validity  of  information,  handling  interruptions,  handling  changes  in  planned 
activity  sequences,  distinguishing  between  modes  of  operation,  gaining  control  over  the  system’s  actions,  and 
identifying  system  errors.  The  systems  also  had  many  problems  related  to  their  general  HFE  design.  Operators 
often  did  not  get  the  information  they  needed;  it  was  presented  in  confusing  formats  not  well  suited  to  their  task 
requirements,  and  excessive  detail  was  given.  This  made  it  difficult  for  operators  to  “visualize  the  intelligent 
system’s  situation  assessment  and  recommendations  in  relation  to  the  flow  of  events  in  the  monitored  process.” 

Similar  results  were  obtained  by  Dien  and  Montmayeul  (1995),  who  surveyed  operating  experience  with  COSSs 
placed  into  existing  CRs.  They  concluded  that  while  much  effort  went  into  their  design,  the  focus  was  on 
technology,  and  feedback  showed  that  operations  were  not  improved  by  their  implementation.  In  many  cases,  the 
approach  led  to  failure.  COSSs  often  provide  guidance  for  situations  that  operators  already  are  equipped  to  handle. 
That  is,  they  are  designed  for  situations  which  were  previously  analyzed  with  which  designers  are  familiar.  Such 
aids  are  of  little  help  to  operators,  except  for  confirmation.  These  systems  poorly  address  unforseen  circumstances 
and  may  not  then  provide  appropriate  guidance. 

Another  problem  observed  was  that  aids  were  “acontextual.”  That  is,  their  guidance  had  little  reference  to  the 
current  situation.  Also,  guidance  was  given  without  appropriately  communicating  what  led  to  its  issuance,  what 
parameters  were  analyzed,  or  what  sequence  of  reasoning  was  followed.  When  the  reasoning  process  is  shown,  it 
may  conflict  with  that  of  the  operators,  i.e.,  it  may  be  based  on  the  designer's  theoretical  understanding  and  not  on 
the  operator's  practical  experience. 

The  new  aids  are  often  poorly  integrated  with  other  HSI  systems,  and  their  design  characteristics,  such  as  dialog 
principles  and  coding,  are  often  different.  Reed,  Hogg,  and  Hallbert  (1995)  found  that  concerns  about  the  interface 
design  and  system  implementation  limited  the  usefulness  of  a  KBS  system,  the  Process  Operations  and 
Management  System,  which  was  installed  in  a  conventional  British  Nuclear  Fuels  plant  to  provide  on-line  early 
warning  and  fault  diagnosis.  These  limitations  led  operators  to  prefer  the  conventional  systems. 

Roth,  Bennet,  and  Woods  (1987)  indicated  that  the  interface  provided  to  a  KBS  must  enable  a  cooperative  dialogue 
so  the  operator  can  better  understand  and  utilize  the  system.  In  general,  these  aids  tend  to  be  technology  driven 
and  do  not  address  the  needs  of  the  operator;  that  is,  they  are  developed  by  finding  an  application  for  a  given 
technology,  rather  than  being  designed  to  meet  users’  needs.  An  inadequate  analysis  of  users’  requirements 
usually  leads  to  problems  with  information  content.  The  system  should  provide  accurate  information  that  is 
needed,  and  not  force  extraneous  material  on  the  user.  Expert  systems  should  support  the  operators’  cognitive 
processes  and  reinforce  their  existing  approach  to  plant  operation  developed  through  training  and  experience.  The 
KBS  should  not  require  operators  to  conform  to  the  machine's  method  of  analysis  (Bernard  and  Washio,  1989). 

KBSs  also  have  inherent  limitations  (Terry,  1989).  They  cannot  reason  broadly  over  a  field  of  expertise,  and  are 
limited  to  narrow  tasks.  They  cannot  reason  from  axioms,  analogies,  or  general  theories.  In  addition,  they  lack 
common  sense  and  often  do  not  make  simplifying  assumptions.  They  are  limited  by  their  programming  and  cannot 
learn.  Their  performance  deteriorates  rapidly  when  applied  to  large  problems.  Such  limitations  of  the  expert 
system  should  be  made  obvious,  so  operators  are  not  required  to  decide  between  their  own  judgment  and  the 
machine- generated  advice  (Bernard  and  Washio,  1989).  The  system’s  security  should  be  controlled  so  that 
inappropriate  changes  cannot  be  made. 
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The  IAEA  (1994)  identified  several  criteria  that  should  be  considered  when  licensing  authorities  evaluate  COSSs: 

( 1 )  Compatibility  with  the  operations  -  To  work  effectively,  operators  may  require  more  than  occasional 
simulator  training  to  become  familiar  enough  with  COSSs  to  use  them.  They  may  need  day-to-day 
experience. 

The  operator’s  effectiveness  in  using  the  COSS  requires  that  the  system  is  used  not  only  in  very 
specific  conditions  for  which  it  was  designed,  but  also  in  normal  operation.  For  mziximum 
compatibility  with  the  global  MMI,  the  data  produced  by  the  COSS  must  be  integrated  into  the 
procedures  used  by  the  operators  for  normal  operations,  as  well  as  in  the  specific  abnormal  or 
emergency  conditions  for  which  the  COSS  was  designed.  This  may  be  an  issue  for  CBP  systems 
that  are  designed  for  emergency  systems  only.  (p.  3 1 ). 

(2)  Consistency  with  the  HSI  design  -  The  detailed  design  of  the  COSS,  e.g.,  labeling  and  dialog  conventions, 
should  be  fully  consistent  with  the  rest  of  the  HSI.  This  may  be  a  particular  problem  for  off-the-shelf 
systems. 

(3)  Cognitive  support  -  The  COSS  must  enhance  performance,  or,  at  a  minimum,  not  degrade  it. 

(4)  Team  performance  -  The  user  of  the  COSS  must  be  clearly  specified.  However,  NPP  control  is  a  team 
operation.  COSSs  may  change  task  allocation  and  the  type  and  quantity  of  information  to  be 
communicated  between  crew  members.  These  effects  must  be  evaluated  and  it  should  be  demonstrated 
that  the  team’s  performance  is  not  degraded. 

Summary 


The  following  human  performance  concerns  are  associated  with  COSSs: 

( 1 )  The  design  of  computer-based  systems  commonly  fails  to  account  for  user  needs.  This  includes  the  need 
for  information  in  the  context  of  current  tasks,  goals,  and  objectives  for  operations,  maintenance,  crew 
configuration,  and  feedback  from  control  actions. 

(2)  COSSs  and  other  computer-based  systems  add  to  the  plant’s  complexity.  Operators  must  have  a  good 
mental  model  or  understanding  of  the  computer-based  system  for  monitoring,  supervision,  and 
maintenance  of  the  plant.  Failure  to  account  for  this  leads  to  poor  situation  awareness  and  a  sense  of 
being  out-of-the-loop. 

(3)  COSSs  often  are  not  designed  so  that  their  logic  is  sufficiently  observable.  That  is,  they  do  not  make  clear 
their  reasoning  basis  or  enable  operators  to  adequately  query  or  otherwise  verify  system  performance. 

(4)  Compatibility  with  day-to-day  operations  needs  to  be  considered.  Systems  with  very  limited  use  in  normal 
operations  that  are  only  used  in  infrequent  special  circumstances  may  have  little  success. 

(5)  Integration  of  a  COSS  with  other  CR  HSIs  is  important. 

(6)  Training  and  team  performance  are  significant  considerations  when  introducing  new  technology. 
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5.6  Summary  and  Discussion 

This  section  summarizes  and  discusses  the  implications  for  CBPs  of  the  material  reviewed  in  Section  5. 

5.6.1  Supervisory  Control  and  Procedure  Use 

The  role  of  the  operating  crew  in  an  NPP  is  that  of  a  supervisory  controller  that  must  engage  in  situation 
assessment,  monitoring  and  detection,  and  response  planning  and  implementation.  These  cognitive  functions  are 
applied  to  tasks  for  which  the  crew  has  primary  manual  responsibility  as  well  as  to  automated  systems,  and  systems 
which  support  the  tasks.  Procedures  fall  into  the  latter  category. 

Historically,  procedures  were  designed  to  support  response  planning  by  providing  operators  with  strategies  based 
on  “off-line”  detailed  analyses  of  both  normal  and  abnormal  states.  However,  when  these  preplanned  strategies  are 
applied  to  the  unique  circumstances  of  a  particular  process  disturbance,  unforseen  or  unanticipated  situations  may 
render  parts  of  a  procedure  inappropriate  or  ineffective.  Thus,  confronted  with  complex,  real-world  process 
disturbances,  operators  must  monitor  the  performance  of  the  procedure  to  verily  its  correspondence  to  the  higher- 
level  goals  that  it  was  designed  to  achieve.  It  is  important  for  operators  then  to  assess  the  effectiveness  of  the 
response  plan  even  when  described  by  established  procedures,  the  consequences  of  particular  actions,  and  the 
appropriateness  of  the  path  for  achieving  identified  goals.  This  enables  operators  to  detect  when  procedures  are 
not  achieving  the  goals,  when  procedures  are  erroneous,  or  when  errors  are  made  in  carrying  out  procedure  steps. 

Another  cognitive  activity  is  adapting  response  plans.  Adapting  plans  to  the  current  situation  is  necessary  because 
steps  may  be  vague  and  have  to  be  interpreted  by  the  operators,  or  their  judgement  is  necessary  to  evaluate  the 
procedure.  In  addition,  procedures  do  not  have  all  the  information  about  the  plant  that  the  operators  do.  Operators 
must  fill  in  the  gaps  in  a  procedure,  modify  it  to  fit  the  specific  situation,  and  direct  the  procedure  path.  Thus, 
rather  than  assuming  the  role  of  rote,  verbatim  procedure  followers,  it  is  important  that  operators  maintain  the  role 
of  supervisory  controllers  and  monitor  the  performance  of  the  procedures  as  well  as  the  process.  They  need  to  stay 
cognitively  involved  in  the  procedure’s  progress.  Operators  need  to  understand  the  intent  of  procedures,  their 
overall  strategies,  assumptions  and  underlying  principles,  and  the  transition  logic  between  procedures.  They 
should  question  procedure  steps  that  appear  inconsistent  with  the  overall  goals  of  the  procedure  for  the  situation  at 
hand. 

With  the  development  of  CBPs,  procedure  systems  have  the  potential  to  support  not  only  response  planning,  but 
also  aspects  of  situation  assessment,  monitoring  and  detection,  and  response  implementation.  This  support  may  be 
applied  to  the  operators’  primary  tasks,  such  as  monitoring  parameters,  and  to  secondary  tasks,  such  as  navigating 
from  one  portion  of  a  procedure  to  another. 

5.6.2  The  Effects  of  CBPs  on  Crew  Performance 

There  are  limitations  to  PBPs  that  CBPs  potentially  can  address:  cognitive  workload  associated  with  process 
monitoring  and  analysis  of  the  logic  in  procedure  steps;  attention  required  for  assessment  of  procedure  steps  that 
are  continuously  applicable,  time  dependent,  and  process  dependent;  the  need  for  varying  levels  of  detail  in 
procedure  information;  the  lack  of  context  sensitivity;  management  and  place  keeping  in  multiple  procedures;  and, 
sequence  control  and  navigation.  The  limitations  of  PBPs  have  been  associated  with  delayed  task  performance  and 
human  errors  in  existing  plants. 
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CBPs  may  address  these  issues;  however,  they  must  maintain  acceptable  performance  on  the  main  tasks  for  which 
procedures  are  used  while  not  introducing  unintended  negative  effects.  The  latter  is  an  important  consideration. 

Our  general  review  of  the  literature  indicated  that  comparisons  of  task  performance  for  information  presented 
either  on  a  VDU  or  in  hard  copy  revealed  significant  differences  between  them.  Reading  on  a  VDU  is  generally 
slower  and  more  fatiguing.  VDU-based  task  performance  also  is  associated  with  slower  and  poorer  performance 
and  concerns  about  usability.  In  addition,  different  task  strategies  are  used.  Contributing  to  these  differences  are 
keyhole  effects,  difficulties  associated  with  maintaining  a  sense  of  location  (knowing  where  you  are  in  a 
document),  navigation  (moving  from  one  place  in  a  document  to  another),  and  fatigue.  Some  of  these  same 
concerns  were  raised  regarding  computerization  of  NPP  procedures  (Chignell  and  Zuberec,  1993). 

In  general,  the  computerization  of  other  types  of  support  systems,  e.g.,  COSSs,  has  had  limited  success  (Dien  and 
Montmayeul,  1995;  IAEA,  1994;  Malin  et  al.,  1991a;  Roth,  Bennet,  and  Woods,  1987).  The  problems  included 
the  failure  to  account  for  users’  needs  and  therefore,  incompatibility  with  day-to-day  operations;  added  complexity 
of  the  HSl;  obscurity  of  the  reasoning  basis;  inadequate  communication  facilities  preventing  operators  from  asking 
questions;  poor  integration  with  other  HSls;  and  personnel  concerns,  such  as  training  and  team  performance. 

Thus,  while  there  are  PBP  deficiencies  that  may  be  resolved  by  computerization,  it  is  essential  to  carefully  examine 
the  effects  on  personnel  performance.  We  did  this  by  reviewing  three  types  of  research:  (1)  empirical  studies  of 
CBPs  where  performance  data  were  collected,  (2)  analyses  of  personnel  performance  using  models,  and  (3)  expert 
opinion  on  postulated  effects  on  performance.  Each  is  briefly  summarized  below. 

The  human  performance  research  was  organized  into  three  categories:  comparisons  of  CBP  and  PBP  systems, 
observations  of  operators'  CBP  use,  and  comparisons  of  design  characteristics  of  procedures.  Several  conclusions 
were  drawn  about  using  CBPs  compared  to  PBPs: 

•  Operators  may  perform  procedure  tasks  more  quickly. 

•  Operators’  cognitive  workload  may  be  reduced. 

•  Operators  may  make  fewer  errors  in  transitioning  through  procedures. 

•  Operators  may  accept  CBPs  more  readily  and  find  them  easier  to  use. 

However,  there  remain  several  important  unresolved  questions  including  (but  not  limited  to)  the  following  ones: 

•  What  is  the  effect  of  CBPs  on  team  performance  and  reliability? 

•  What  is  the  effect  of  CBPs  on  operators’  high-level  situation  awareness  of  the  status  of  the  overall  procedure 
goal  and  the  plant? 

•  Do  operators  become  overreliant  and  unquestioning  of  CBPs,  or  can  they  maintain  independence  and 
objectivity  to  evaluate  the  adequacy  of  the  procedure  to  achieve  its  goals? 

•  What  is  the  effect  of  CBPs  on  performance  in  complex  disturbances  that  may  involve  many  procedures  or 
branches? 

•  How  well  do  operators  manage  complex  CBP  failures,  such  as  when  multiple  procedures  are  being  used,  many 
steps  have  been  completed,  and  many  steps  of  continuous  applicability  are  being  monitored? 
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•  What  are  the  relative  effects  of  specific  design  features  on  performance? 

These  and  other  CBP  issues  are  discussed  in  Section  5.6.3. 

Another  problem  with  the  human  performance  research  reviewed  was  that  many  studies  were  not  discussed  in 
sufficient  detail  to  evaluate  their  generalizability.  Those  studies  that  were  sufficiently  documented  had  potential 
methodological  weaknesses  which  limited  their  conclusions  and  generalizability. 

Personnel  performance  also  was  analyzed  using  two  classes  of  analytical  techniques:  performance  models  and  risk 
models,  CBPs  were  evaluated  by  a  variety  of  performance  analysis  models  including  the  GOMS  model, 

MicroSaint  Task  Network  Modeling,  the  MIDAS  model,  and  classical  task  analysis.  The  performance  models 
showed  CBPs  had  no  clear  advantage  over  PBPs.  Instead,  they  illustrated  the  importance  of  performance  tradeoffs 
in  assessing  different  procedure  systems.  In  general,  complexity  and  attentional  demands  were  higher  for  CBPs 
while  data  retrieval  was  easier  and  task  completion  time  was  less. 

It  is  interesting  that  the  use  of  performance  models  for  evaluating  procedures  has  had  some  success.  Their 
continued  development  may  focus  the  testing  of  specific  design  issues  that  must  be  addressed  in  CBP  design;  this 
view  is  consistent  with  the  conclusions  of  a  National  Academy  of  Science  (NAS)  assessment  of  applying 
quantitative  models  of  human  performance  to  complex  systems.  The  NAS  (Baron  et  al.,  1990)  concluded  that  “In 
all,  there  are  compelling  reasons  to  believe  that  systematic  human  performance  modeling  efforts  should  be 
regularly  advocated  and  used  along  with  expert  judgement  and  manned  part-  and  full-task  simulation,  as  a  regular 
part  of  the  design  process  for  large-scale  human-machine  systems”  (p.  86). 

Several  risk-informed  analyses  of  CBPs  have  been  made,  each  looking  at  risk  somewhat  differently:  examining  the 
potential  to  change  generic  failure  rates,  the  potential  effects  of  CBPs  on  components  of  human  error  probabilities, 
and  whether  CBPs  may  represent  a  potentially  safety  significant  issue. 

Like  the  studies  with  performance  model  analyses,  the  findings  were  mixed.  They  illustrated  the  f)Otential  for  these 
systems  to  improve  performance  by  supporting  such  procedure-related  work  as  process  monitoring,  logic  analysis, 
navigation,  and  place  keeping.  However,  they  indicated  that  poorly  implemented  CBPs  can  reduce  human 
reliability. 

Finally,  SME  opinion  on  the  postulated  effects  on  personnel  performance  was  reviewed,  including  an  NRC- 
sponsored  review  of  CBP  systems,  an  NRC  CBP  workshop,  and  an  IAEA  working  group  on  computerization  of 
CRs  that  addressed  CBPs.  The  SME  review  of  CBPs  identified  many  positive  effects  of  their  use  on  the  crew’s 
performance;  however,  a  wide  range  of  issues  was  identified  to  be  resolved  in  developing  CBPs.  Also  noteworthy 
was  the  observation  that  CBP  development  must  consider  related  HFE  activities,  such  as  training,  and  integrating 
the  CBP  system  with  the  other  HSIs  and  with  the  plant’s  overall  operational  philosophy.  V&V  programs  were 
emphasized.  In  general,  these  findings  were  consistent  with  those  from  other  sources  discussed  earlier. 

Considering  all  the  results,  we  concluded  that  CBPs  have  the  potential  to  support  operators'  performance  and  there 
is  evidence  to  support  this  claim.  As  the  NRC  indicated  in  its  review  of  the  URD,  “...the  development  of 
electronically  displayed  procedures  is  a  desirable  goal  for  the  overall  integration  of  operator  information  needs” 
(NRC,  1994). 

However,  there  are  also  important  issues  to  be  considered  both  in  research  and  in  the  development  of  individual 
systems.  Thus,  the  advice  of  several  researchers  and  developers  of  CBP  systems  is  repeated:  the  development  of 
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CBP  systems  for  operational  use  should  proceed  in  a  way  that  the  benefits  and  drawbacks  of  CBP  systems  can  be 
fully  evaluated  for  each  specific  system.  CBPs  have  important  impacts  on  NPP  operations,  some  of  which  extend 
beyond  those  intended  by  the  designers  (Spurgin,  Wachtel,  and  Moieni,  1993). 

The  following  are  some  general  considerations  for  near-term  approaches  to  CBP  systems: 

•  Support  cognitive  functions  which  may  be  distracting  and  error  prone,  such  as 

process  monitoring 

logic  analysis  (cautiously,  so  as  not  to  underspecify  the  analysis  and  undermine  the  operator’s  judgement) 

•  Support  procedure  management,  e.g.,  step  completion,  place  keeping,  transitioning  between  procedures 

•  Provide  PBP  backup  systems  and  ensure  the  similarity  of  CBPs  and  PBPs  to  (1)  ensure  confidence  in  near- 
term  CBP  applications,  (2)  enable  operating  experience  to  be  gained,  (3)  minimize  impacts  on  function 
allocation,  (4)  reduce  burdens  in  training  operators  to  use  both  systems,  and  (5)  ensure  successful  crew 
performance  when  using  backups  (minimize  the  potential  for  negative  transfer  or  difficulties  in  performance 
arising  from  disuse). 

5.6.3  CBP  Issues 

This  section  summarizes  the  human  performance  issues  associated  with  CBPs  identified  from  the  literature  review. 
These  issues  represent  topics  for  which  research  is  needed  before  developing  additional  guidance.  From  a 
regulatory  review  perspective,  these  issues  may  be  addressed  on  a  case-by-case  basis,  as  part  of  the  design  process 
review  discussed  in  Part  2  of  this  document. 

The  issues  are  not  mutually  exclusive;  they  overlap  and  some  are  more  general  than  others;  some  may  be 
considered  secondary  to  others.  Interdependencies  are  unavoidable,  as  they  all  pertain  to  the  interactions  within  an 
integrated  human-machine  system. 

Methodological  and  Criterion  Requirements  for  Evaluating  CBP  Effects 

A  more  definitive  conclusion  about  the  value  of  CBP  systems  was  hampered  by  the  lack  of  operational  experience 
with  their  use,  and  lack  of  quality  experimental  evaluations.  The  detailed  methodological  considerations  validating 
complex  human-machine  systems  and  a  conceptual  approach  to  it  were  discussed.  The  methodology  focused  on 
(1)  establishing  the  requirements  for  making  a  logical,  defensible  inference  from  validation  tests  to  predicted 
integrated  system  performance  under  actual  operating  conditions,  and  (2)  identifying  aspects  of  validation 
methodology  that  are  important  to  the  inference  process.  The  technical  basis  for  inference  in  validation  is  based 
upon  four  general  forms  of  validity:  system  representation,  performance  representation,  test  design,  and  statistical 
conclusion. 

The  studies  examined  generally  had  not  undertaken  well-controlled,  comprehensive  evaluations  that  would  supply 
valuable  data  to  better  understand  the  impact  of  CBP  effects  under  a  wide  range  of  scenarios  and  complex 
situations,  using  varied  personnel  and  system  measures.  Most  of  them  had  methodological  weaknesses  which 
limited  their  conclusiveness  and  generalizability.  Thus,  important  questions  remain  (many  are  discussed  in  more 
detail  below).  A  good  comprehensive  evaluation  of  CBPs  and  their  effects  on  crew  performance  has  yet  to  be 
made. 
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One  question  that  needs  to  be  addressed  from  both  research  and  regulatory  review  perspectives  is,  “What  are  the 
criteria  for  CBP  acceptance?”  While  some  authors  specified  that  such  systems  should  improve  performance,  others 
indicated  that  performance  should  not  be  degraded  (implying  that  equivalent  performance  with  PBPs  and  CBPs  is 
acceptable).  This  is  an  extremely  important  distinction  because  of  the  impact  on  performance  that  would  be 
necessary  if  CBPs  were  required  to  improve  it. 

Role  of  Plant  Personnel  in  Managing  Procedures 

Procedures  are  guidance  to  operators  for  achieving  high-level  objectives.  While  they  are  correct  most  of  the  time, 
for  analyzed  situations,  adaptation  sometimes  may  be  necessary.  Thus,  operators  must  remain  as  indep>endent 
supervisors  who  manage  procedure  implementation  and  independently  assess  its  appropriateness.  Operators  must 
understand  the  overall  purpose  of  the  procedures,  stay  cognitively  involved  with  their  progress,  and  question  any 
steps  inconsistent  with  the  overall  high-level  safety  goals.  However,  CBPs  potentially  might  work  against  this 
independence,  minimizing  the  operator’s  role.  They  may  increase  the  tendency  to  follow  procedures  without  a 
critical  independent  persp)ective,  and  may  even  deter  the  operator’s  action.  Resolving  these  concerns  affects  both 
design  and  training. 

Thus,  one  pressing  issue  is  how  to  design  and  review  CBP  systems  that  enable  the  operators  to  maintain  this 
independent  perspective,  but  at  the  same  time,  reduce  the  operator’s  workload,  automate  distracting  and  lower- 
level  error-prone  tasks,  and  monitor  the  crew’s  performance,  especially  when  the  crew  and  CBPs  disagree.  Equally 
important  is  how  to  train  operators  in  handling  this  role  while  using  CBPs.  The  knowledge  required  to  manage  a 
CBP  system  may  differ  from  that  required  to  handle  PBPs.  For  example,  the  CBP  system  may  use  different 
analyses  to  resolve  procedure  logic  steps  than  operators  do. 

Team  Performance 


Research  showed  that  CBPs  may  significantly  affect  crew  member’s  roles,  teamwork,  and  communication. 
Teamwork  is  an  important  element  of  defense-in-depth.  Operators  work  as  a  team  to  support  situation  assessment, 
error  detection  and  recovery.  These  roles  and  communication  may  be  changed  more  than  anticipated.  Since  SROs 
using  CBPs  can  handle  a  procedure  almost  completely  on  their  own,  communication  between  the  SRO  and  ROs 
may  be  reduced  (Roth  and  O'Hara,  1998).  While  this  is  not,  in  itself,  good  or  bad,  its  impact  on  team  performance 
needs  assessment.  Board  operators  identified  the  importance  of  communication  in  maintaining  effective  teamwork 
when  the  SRO  is  using  a  CBP  and  expressed  a  need  to  be  aware  of  the  status  of  EOPs.  Thus,  the  potential  for 
isolating  the  CBP  user  from  the  other  operators,  and  changing  operators’  roles  and  responsibilities  may  undermine 
team  performance  in  emergencies.  Such  effects  on  team  performance  were  noted  for  many  aspects  of  computer- 
based  HSl  technology  (Stubler  and  O'Hara,  1996). 

The  function  of  supporting  coordination  of  the  crew’s  work  centers  on  the  need  for  operators  to  be  aware  of  the 
activities  of  other  crew  members.  The  CR  is  the  context  within  which  personnel  convey,  directly  and  indirectly, 
their  intentions  and  actions  to  others.  Advanced  CRs,  especially  those  with  individual  workstations,  may  isolate 
operators,  making  an  individual’s  information  and  control  actions  less  visible  to  others,  thus  reducing  team 
effectiveness. 

Salas  et  al.  (1992)  define  a  team  as  “...a  distinguished  set  of  two  or  more  people  who  interact,  dynamically, 
interdep)endently,  and  adaptively  toward  a  common  and  valued  goal/objective/mission,  who  have  each  been 
assigned  specific  roles  or  functions  to  perform”  (p.  4).  In  a  CR  setting,  operators  must  share  information  and 
coordinate  their  tasks  to  satisfy  specific  goals  or  mission  requirements.  This  requires  a  common  understanding  of 
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the  status  of  the  system  and  of  each  others'  actions  and  intentions.  Identification  and  resolution  of  errors, 
coordinated  information  exchange,  and  team  reinforcement  were  identified  as  important  to  team  performance 
(Oser  et  al.,  1 989).  Successful  teams  actively  located  errors,  questioned  improper  procedures,  and  monitored  the 
status  of  others.  In  a  study  of  ship  navigation,  Hutchins  (1990)  discussed  team  performance  in  terms  of  facilitating 
error  checking  by  others,  allowing  others  to  assist,  and  supporting  training  in  the  work  setting. 

Hutchins  found  that  longstanding  work  environments  with  traditional  technologies  have  characteristics  that 
contribute  to  team  performance:  horizon  of  observation,  openness  of  tools,  and  openness  of  interaction.  However, 
when  computer-based  technologies  are  introduced,  these  positive  characteristics  may  be  compromised. 

•  Horizon  of  Observation  -  This  refers  to  the  portion  of  the  team  task  that  can  be  seen  or  heard  by  each 
individual.  It  results  from  the  arrangement  of  the  work  environment  (e.g.,  proximity  of  team  members)  and  is 
influenced  by  the  openness  of  tools  and  interactions.  By  making  portions  of  a  task  more  observable,  team 
members  can  monitor  errors  of  intent  and  implementation,  and  determine  when  assistance  might  be  helpful. 

•  Openness  of  Tools  -  This  is  the  degree  to  which  an  observer  is  able  to  infer  information  about  another’s 
ongoing  tasks  through  observation  of  a  tool's  use.  Open  tools  show  characteristics  of  the  problem  that  give  an 
observer  the  context  for  understanding  what  has  been  done  and  the  possible  implications. 

•  Openness  of  Interaction  -  This  is  the  degree  to  which  the  interactions  between  team  members  provide  an 
opportunity  for  others  with  relevant  information  to  contribute.  Openness  of  interaction  depends  on  the  type  of 
communication  (e.g.,  discussing  actions  or  decisions  in  the  presence  of  others)  and  the  style  of  interaction 
(e.g.,  the  extent  to  which  unsolicited  input  is  accepted).  Openness  of  interaction  is  also  influenced  by 
characteristics  of  the  work  environment  (e.g.,  openness  of  tools,  horizon  of  observation)  that  allow  other  team 
members  to  see  and  hear  the  interaction. 

Conventional  CR  designs  typically  have  a  broad  horizon  of  observation  facilitating  the  observation  of  team 
activities.  In  addition,  they  may  be  “open  tools”  in  the  sense  that  an  observer  can  infer  information  about  control 
actions  (e.g.,  which  plant  system  was  involved,  which  control  was  operated,  and  what  action  was  taken)  by 
observing  the  operator's  location  at  a  control  panel  and  the  action  performed.  Interactions  may  be  considered 
“open”  because  most  involve  speech  that  can  be  heard  from  across  the  CR. 

Advanced  HSl  technologies,  such  as  CBPs,  may  impair  these  good  characteristics.  For  example,  using  an 
individual  computer-based  workstation  with  an  individual  view  of  the  plant  may  reduce  the  horizon  of  observation 
because  that  view  cannot  be  readily  seen  by  others  and  may  lead  to  less  open  styles  of  communication.  Also,  the 
openness  of  tools  may  be  impaired  by  having  methods  of  user-system  interaction  that  convey  less  task-related 
information  to  observers. 

Situation  Assessment,  Response  Planning,  and  Operator  Error 

The  effect  of  CBPs  on  the  operator’s  situation  awareness  has  not  been  carefully  evaluated.  Operators  need  to 
maintain  several  levels  of  situation  awareness  when  using  procedures,  including  assessment  of: 

•  procedure  steps,  how  procedures  are  structured,  one's  location  within  a  procedure  or  between  a  set  of 
procedures, 

•  the  appropriateness  of  procedures  to  achieve  high-level  procedure  goals,  and 
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•  the  overall  situation  in  the  plant 

Some  concern  over  lowered  situation  awareness  with  CBPs  was  noted  (Roth  and  O'Hara,  1998).  PBPs  require 
operators  to  monitor  plant  indications.  However,  if  CBPs  are  present,  the  operator  may  not  feel  the  need  to  look  at 
other  sources  of  information  and  may  miss  important  indications  that  are  not  present  in  the  CBP.  The  situation 
awareness  of  other  operators  is  affected  as  well.  Spurgin  et  al.  (1990)  noted  that  SROs  use  CBPs  as  their  primary 
way  of  following  the  overall  plant  condition  rather  than  relying  on  information  from  crew  members. 

Consequently,  the  other  crew  members  expressed  concern  about  being  aware  of  the  EOP  status. 

Jeffroy  and  Charron  (1997)  described  another  aspect  of  situation  awareness  -  the  joint  awareness  of  the  operators 
and  the  CBP.  Such  combined  awareness  may  be  separated  when  operators  depart  from  the  recommendations  of  the 
CBP,  creating  a  situation  that  makes  it  difficult  for  them  to  recognize  the  CBP’s  constraints.  They  may  not 
understand  how  their  actions  affect  the  procedure’s  ability  to  analyze  individual  steps. 

Research  is  needed  to  clarify  the  effects  of  CBPs  on  these  different  levels  of  situation  awareness,  the  crew’s  ability 
to  detect  errors,  and  adaptation  of  the  response  plan  in  the  face  of  procedure  failures.  In  addition,  knowledge  is 
needed  on  the  effect  of  CBPs  on  the  number  and  types  of  operator  errors  (especially  where  errors  are  not  defined  in 
terms  of  verbatim  compliance). 

Two  related  issues  affect  situation  awareness:  complexity  and  level  of  abstraction.  Research  on  COSSs 
emphasized  that  computerized  support  systems  add  to  complexity.  Operators  need  a  good  mental  model  or 
understanding  of  the  computer-based  system  to  properly  monitor  and  supervise  the  CBP.  Failure  to  account  for 
this  leads  to  poor  situation  awareness  and  a  sense  of  being  out-of-the-loop. 

Roth  and  O'Hara  (1998)  observed  that  too  little  information  presented  at  each  procedure  step  can  cause  operators  to 
lose  a  sense  of  where  they  are,  while  too  much  may  distract  them.  The  level  of  abstraction  at  which  the  results  of 
procedure  steps  are  presented  will  affect  the  operators’  situation  assessment. 

Level  of  Automation  of  Procedure  Functions 


The  human  performance  issues  associated  with  automation  have  been  well  documented;  see  O'Hara,  Stubler,  and 
Higgins  (1996)  for  a  discussion  of  general  automation.  Table  4.1  listed  procedure-related  functions  in  terms  of 
several  levels  of  automation.  The  choices  of  levels  of  automation  and  their  implementation  will  impact  operator’s 
performance,  situation  awareness,  workload,  and  errors.  Blackman  and  Nelson  (1988)  found  that  when  the 
procedures  were  selected  automatically,  operators’  involvement  was  reduced;  they  reported  that  they  thought  less 
and  acted  as  switch-turners.  A  better  understanding  is  needed  of  the  tradeoffs  between  automatic  procedure 
functions  and  operators’  involvement,  independence,  and  supervisory  control. 

One  area  of  procedure  automation  is  especially  noteworthy.  An  important  capability  of  CBP  systems  is  the  analysis 
of  procedure  step  logic;  that  is,  comparing  actual  parameter  values  to  the  reference  value  in  procedures  using  the 
logical  relationships  described  in  the  step.  When  the  step  logic  or  the  data  analysis  required  to  evaluate  the  step 
logic  is  underspecified,  both  the  procedure  and  the  operator  can  misjudge  the  situation.  Therefore,  procedures, 
especially  EOPs,  must  be  carefully  designed  and  evaluated  to  guard  against  such  underspecification.  Where  the 
operator’s  judgement  is  involved,  such  analyses  are  better  kept  manual. 
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Keyhole  Effects  and  Use  of  Multiple  CBPs 

Viewing  information  through  the  limited  area  provided  by  VDUs  is  referred  to  as  the  “keyhole  effect”  (Woods  et 
al.,  1990);  its  consequence  is  that,  at  any  time,  most  information  is  hidden.  Therefore,  operators  must  know  what 
information  and  controls  are  available  in  the  computer  system,  where  they  are,  and  how  to  navigate  and  retrieve 
them. 

The  keyhole  effect  was  identified  as  a  root  cause  of  many  challenges  to  performance  (O'Hara,  Stubler,  and  Nasta, 
1997).  If  the  viewing  area  is  insufficient  for  operators  to  perform  their  tasks,  they  may  have  to  navigate  repeatedly. 
A  problem  with  the  keyhole  effect  is  that  access  to  controls  and  displays  tends  to  be  serial,  e.g.,  only  a  few  controls 
can  be  accessed  at  once,  in  contrast  to  the  parallel  presentation  of  controls  and  displays  in  conventional  CRs.  The 
sheer  burden  of  navigating  and  retrieving  many  displays  can  interfere  with  the  operators’  ability  to  obtain  an 
overview  of  the  plant’s  situation.  If  workload  is  already  high,  operators  decide  not  to  retrieve  all  the  information 
they  need  so  they  can  invest  their  mental  resources  in  their  current  task. 

The  issues  may  become  significant  for  CBPs  when  operators  are  required  to  be  in  multiple  procedures.  Hoecker 
et  al.  (1994)  and  Hoecker  and  Roth  (1996)  noted  that  when  the  operators  are  required  to  obtain  information  in 
parallel,  the  CBP  system  can  increase  workload.  This  lack  of  parallel  access  is  a  limitation  of  the  keyhole  effect. 
Because  only  a  portion  of  the  procedure  can  be  observed  in  the  display’s  space  at  one  time,  operators  may  lose  a 
sense  of  where  they  are  within  the  total  set  of  active  procedures. 

CBP  Failure  in  Complex  Situations 

Ensuring  the  transfer  from  CBPs  to  PBPs  was  recognized  as  an  important  consideration  in  designing  and 
evaluating  CBPs.  This  transition  may  be  easy  when  the  procedure  context  is  simple,  such  as  when  operators  are  in 
its  first  few  steps.  However,  the  transition  may  be  quite  complex  if  operators  are  deep  into  the  procedures,  or  when 
there  are  multiple  procedures  open,  many  steps  completed,  many  steps  of  continuously  applicability,  time- 
dependent  steps,  and  parameter-dependent  steps  being  monitored  by  the  CBPs.  How  operators  will  manage 
failures  in  such  complex  situations  is  unknown. 

Hybrid  Procedure  Systems 

Some  CBP  systems  computerize  all  plant  procedures  (e.g.,  in  EdF  N4  CBP)  while  others  contain  only  the  EOPs 
(e.g.,  EOPTS).  The  ability  to  use  CBPs  effectively  when  they  are  designed  only  for  emergencies  may  be 
problematic. 

Several  studies  recommended  that  for  COSSs  to  be  effective,  they  must  be  well  integrated  into  everyday  operations. 
Further,  operators  may  require  more  than  occasional  simulator  training  to  become  familiar  enough  with  COSSs  to 
use  them,  as  the  IAEA  (1994)  stated: 

Operator  effectiveness  in  using  the  COSS  requires  that  the  system  be  used  not  only  in  very  specific 
conditions  for  which  it  was  designed  but  also  in  normal  operation.  For  maximum  compatibility  with 
the  global  MMI,  it  is  necessary  to  integrate  the  data  produced  by  the  COSS  into  the  procedures  used 
by  the  operators  for  normal  operations,  as  well  as  in  the  specific  abnormal  or  emergency  conditions 
for  which  the  COSS  may  have  been  designed.  Note  that  this  may  be  an  issue  for  CBP  systems  that 
are  designed  for  emergency  systems  only.  (p.  31) 
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Others  commented  on  the  need  for  CBPs  to  be  consistent  with  normal,  daily  operations  (NRC  CBP  Workshop). 
While  EOPs  are  not  used  in  daily  operations,  their  computerization  and  the  use  and  functionality  of  the  system  may 
raise  difficulties  if  the  operators’  interactions  with  them  are  unlike  those  with  other  systems  in  the  CR. 

Specific  CBP  Design  Features 

Most  studies  we  reviewed  did  not  discuss  the  relative  effects  of  specific  CBP  design  features  on  performance.  They 
were  overall  system  comparisons,  e.g.,  CBP  vs.  PBP,  not  systematic  evaluations  of  individual  characteristics.  In 
addition,  concern  over  the  generalization  of  PBP  guidance  to  CBPs  was  expressed. 

Thus,  traditional  procedure  formats  may  require  modifications  when  implemented  on  a  computer.  Two  primary 
formats  are  used  for  procedures:  text  and  flowcharts  (Section  4).  While  both  are  successful  in  paper  form, 

Chignell  and  Zuberec  (1993)  questioned  whether  flowchart  presentations  are  acceptable  in  computer  media  where 
the  limited  screen  and  need  for  scrolling  may  make  them  less  effective.  Similarly,  reading  extended  text  from 
VDUs  was  found  to  be  visually  fatiguing.  The  proper  implementation  of  CBPs  in  text  and  flowchart  formats  may 
require  more  guidance  than  that  available  for  the  paper  forms.  The  effects  of  HSI  techniques  (such  as  outline 
views,  navigational  aids,  and  highlighting)  on  text  and  flowchart  use,  requires  exploration. 
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Section  5.6.2  identified  an  approach  to  near-term  CBPs,  based  on  a  consideration  of  lessons  learned  and  of  the 
remaining  CBP  issues.  General  considerations  for  near-term  approaches  to  CBP  systems  include  (1 )  supporting 
cognitive  functions,  such  as  process  monitoring  and  logic  analysis;  (2)  supporting  procedure  management,  such  as 
step  completion,  place  keeping,  and  transitioning  between  procedures;  and  (3)  providing  PBP  backup  systems  and 
ensuring  similarity  of  CBPs  to  them.  The  review  reflects  these  considerations. 

As  an  emerging  technology,  the  technical  basis  for  CBP  guidance  is  limited,  and  there  remain  unresolved  issues 
that  cannot  currently  be  reviewed  with  HFE  guidelines.  Thus,  CBP  reviews  will  require  review  of  the  CBP  design 
processes  as  well  as  the  design  implementation.  The  latter  conclusion  is  fully  consistent  with  the  NUREG-0700 
approach  to  reviewing  HSl  technology  for  which  HFE  design  review  guidance  is  limited,  and  also  with  the  URD 
approach  to  an  unproven  technology.  The  development  of  guidance,  therefore,  took  two  forms:  HFE  design 
review  guidance  (typically  found  in  NUREG-0700),  and  design  process  guidance.  Guidance  for  products  and 
processes  is  described  in  Part  2  of  this  report. 

As  part  of  the  design  process,  CBP  systems  will  have  to  be  evaluated  using  simulations  and  comparisons  to 
reference  systems,  i.e.,  PBPs.  This  will  confirm  their  acceptability  and  will  support  the  development  of  more 
detailed  guidance  for  specific  systems.  In  the  staffs  URD  review,  the  NRC  (1994)  stated  that  the: 

...  designer  should  consider  the  use  of  electronically  displayed  procedures  early  in  the  design  process  to  resolve 
any  issues  concerning  their  development,  operability,  maintainability,  and  reliability.  If  electronically  displayed 
procedures  are  determined  to  be  an  improvement  over  hard-copy  procedures  and  the  M-MIS  designer  has 
integrated  electronically  displayed  procedures  into  the  overall  M-MIS  design,  they  should  be  provided  as  part 
of  the  design,  (p.  lO.B-17) 

NUREG-071 1  gives  the  NRC’s  high-level  design  process  criteria  for  reviewing  overall  HFE  programmatic  goals 
and  objectives  but  not  for  detailed  reviews  of  final  HSl  designs,  such  as  displays,  control,  or  procedures.  Rather  it 
cites  NUREG-0700  for  guidance  on  detailed  reviews  of  plant  HSIs.  In  Pcirt  1  of  NUREG-0700,  Rev.  1,  the  design 
process  is  covered  under  the  review  of  plant  HSIs  for  which  the  general  framework  of  NUREG-071 1  was  used. 
However,  Part  1  also  addresses  general  aspects  of  reviewing  HSl  designs,  i.e.,  it  does  not  identify  the  specific 
considerations  that  may  be  important  for  an  individual  technology,  such  as  CBPs.  The  only  detailed  HSl 
technology-specific  guidelines  are  in  Part  2  of  NUREG-0700;  they  cover  the  detailed  form  and  functional 
characteristics  for  HSl  implementations,  but  not  design  process  considerations. 

Both  types  of  guidance  are  necessary  for  a  design  review  of  CBPs.  That  is,  while  there  is  sufficient  technical  basis 
to  develop  detailed  design-implementation  guidance  for  many  characteristics  of  CBPs,  as  is  typical  in 
NUREG-0700,  several  limitations  in  the  technical  basis  have  been  identified,  and  so  issues  remain  for  which 
typical  NUREG-0700  guidelines  could  not  be  developed.  However,  until  more  guidance  becomes  available,  these 
issues  can  be  resolved  for  specific  CBP  systems.  To  support  resolution  of  issues  for  specific  systems,  guidance  for 
the  CBP  design  process  review  was  established. 

6.1  CBP  Design  Process  Review  Guidance 

Guidelines  were  formulated  for  the  design  process  review  to  address  important  points  raised  in  the  literature,  and 
to  provide  a  place  where  CBP  human  performance  issues  could  be  explored  during  a  design  review  on  a  case-by¬ 
case  basis.  The  format  of  the  guidelines  corresponds  to  the  NRC*s  general  guidance  in  NUREG-071 1 .  They  are 
organized  into  the  following  sections: 
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•  HFE  Program  Management 

•  Operating  Experience  Review 

•  Functional  and  Task  Analysis 

•  Staffing 

•  Human  Reliability  Analysis 

•  Human-System  Interface  Design 

•  Procedure  Development 

Scope 

Bases 

Technical  Information 
Maintenance 

•  Training  Program  Development 

•  Human  Factors  Verification  and  Validation 

These  guidelines  are  discussed  in  Section  9,  Part  2  of  this  report. 

6.2  CBP  HFE  Design  Review  Guidelines 

A  draft  set  of  guidelines  was  developed  from  the  findings  and  source  materials  that  we  surveyed,  along  with  the 
high-level  design  review  principles  from  NUREG-0700,  Rev.  1.  These  principles  were  developed  from  research 
and  industrial  experience  on  integrating  personnel  into  complex  systems.  They  reflect  the  important  design  goals 
of  (I)  maximizing  primary  task  performance  (i.e.,  process  monitoring,  decision  making,  and  control),  (2) 
minimizing  secondary  task  demands  unrelated  to  the  primary  task  (e.g.,  the  distraction  of  tasks  such  as  configuring 
a  workstation),  and  (3)  minimizing  human  errors  and  making  systems  more  tolerant  of  them.  These  principles  are 
also  set  out  in  Appendix  B. 

An  example  of  these  guidelines  written  in  the  standard  format  of  NUREG-0700,  Rev.  1  is  presented  below: 

10.2.2-2  Automatic  Monitoring  of  Plant  Parameters  and  Equipment  Status 

The  CBP  should  automatically  provide  accurate  and  valid  information  on  the  values  of  parameters  and 

status  of  equipment,  when  they  are  available  to  the  system. 

ADDITIONAL  INFORMATION:  It  should  be  clear  to  operators  what  specific  information  is  used  as  the 
source  of  these  actual  values  and  states. 

Discussion:  Supporting  cognitive  functions,  such  as  obtaining  parameter  values  (monitoring)  may  reduce 
the  demands  on  attentional  resources  and  working  memory  and  enable  the  operator  to  focus  more  on 
evaluating  higher-level  procedure  goals.  It  may  also  help  solve  PBP  issues.  This  capability  was  identified 
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as  being  beneficial  to  the  crew’s  reliability  (Orvis  and  Spurgin,  1 996;  Pirus  and  Cham  bon,  1 997;  Niwa  et 
al.,  1996).  Further,  presenting  plant  parameters  and  status  in  procedure  steps  is  a  URD  requirement 
(EPRl,  1 993a).  This  guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Situation 
Awareness  and  Cognitive  Workload  (see  Appendix  B). 

Each  guideline  has  the  following  components: 

•  Guideline  Number  -  Within  each  section,  individual  guidelines  are  numbered  consecutively.  The  number 
includes  its  section  and  subsection  location,  followed  by  a  dash  and  its  unique  number. 

•  Guideline  Tide  -  Each  guideline  has  a  brief,  unique,  descriptive  title. 

•  Review  Criterion  -  Each  guideline  contains  a  statement  of  an  HSl  characteristic  so  that  the  reviewer  may  judge 
the  HSl's  acceptability.  The  criterion  is  not  a  requirement,  and  discrepant  characteristics  may  be  judged 
acceptable  based  on  the  procedures  in  the  review  process. 

•  Additional  Information  -  For  many  guidelines,  there  is  additional  information  including  clarifications, 
examples,  exceptions,  details  on  measurement,  figures,  and  tables  to  support  the  reviewer’s  interpretation  or 
application  of  the  guideline. 

•  Discussion  -  The  discussion  summarizes  the  technical  basis  of  the  guideline.  It  may  identify  the  primary 
source  documents,  the  technical  literature,  such  as  journal  articles,  or  the  general  principles  from  which  the 
guideline  was  derived.  This  section  will  be  removed  when  the  guidance  is  integrated  into  NUREG-0700, 

Rev.  2. 

In  place  of  the  Discussion  will  be  a  Source  field: 

•  Source  -  The  source  field  identifies  the  NUREG  or  NUREG/CR  (or  other  document)  containing  the  technical 
basis  and  development  methodology  for  the  guideline.  As  is  the  standard  practice  for  NUREG-0700,  the  source 
field  will  give  a  reference  to  this  document. 

The  guidelines,  contained  in  Section  10,  are  organized  into  the  following  sections: 

•  Procedure  Representation 

Identification 
Basic  Steps 

Warnings,  Cautions,  Notes,  and  Reference  Materials 
Lists 

Organization 


Formatting  and  Screen  Layout 
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•  Procedure  Functionality 

Procedure  Supervision  and  Control 
Procedure  Monitoring  and  Assessment 
Monitoring  Operator  Actions 
Planning  and  Implementation 

•  Procedure  Management  and  Support 

Path  Monitoring 
Navigation 

Communication  and  Help 

•  CBP  Hardware 

•  CBP  Procedure  Backup 

•  CBP  Integration  with  Other  HSI  Components. 


NUREG/CR-6634 


6-4 


7  SUMMARY 


The  objective  of  this  study  was  to  develop  HFE  review  guidance  for  CBP  systems  based  on  a  technically  valid 
methodology.  To  support  this  objective,  the  following  tasks  were  undertaken: 

•  Development  of  a  framework  for  characterizing  key  design  features  of  CBP  systems 

•  Development  of  a  technical  basis  using  research  and  analyses  on  human  performance  relevant  to  CBPs 

•  Development  of  HFE  review  guidelines  for  CBPs  in  a  format  that  is  consistent  with  NURBG-0700,  Rev.  1 ,  and 
NUREG-0711 

•  Identification  of  remaining  CBP  issues  for  which  research  was  insufficient  to  support  our  development  of  NRC 
review  guidance 


The  status  of  each  will  be  briefly  addressed  below. 

CBP  System  Characterization  Framework 

For  this  study,  CBP  systems  were  narrowly  defined  to  encompass  computer  systems  that  support  procedure 
presentation  and  use.  The  focus  was  on  the  HFE  aspects  of  CBPs,  and  not  the  l&C  or  software  aspects  (although 
the  latter  are  important  as  well,  and  are  described  in  other  NRC  regulatory  and  research  programs).  CBPs  were 
characterized  along  the  following  the  dimensions: 

•  Representation  of  Procedure  Elements 

•  Procedure  Functionality 

•  Interface  Management  and  Support 

•  CBP  Hardware 

•  Backup  Systems  for  Procedures 

•  Integration  of  CBP  System  with  the  HSl 
Development  of  the  Technical  Basis 

The  effects  of  CBPs  on  crew  performance  were  determined  by  examining  three  types  of  research:  (1)  empirical 
studies  of  CBPs  where  data  on  personnel  performance  were  collected,  (2)  analyses  of  personnel  performance  using 
models,  and  (3)  expert  opinion  about  their  postulated  effects  on  personnel  performance. 

The  human  performance  research  was  organized  into  three  categories:  comparisons  of  CBP  and  PBP  systems, 
observations  of  operators’  use  of  CBPs,  and  comparisons  of  design  characteristics  of  procedures.  Several 
conclusions  were  made  from  comparing  CBPs  with  PBPs: 

•  Operators  perform  tasks  more  quickly. 
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•  Operators’  overall  cognitive  workload  is  reduced. 

•  Operators  may  make  fewer  errors  in  transitioning  through  procedures. 

•  Operators  may  accept  CBPs  readily  and  find  them  easier  to  use. 

However,  much  of  the  human  performance  research  had  insufficient  detail  to  evaluate  its  generalizability.  Studies 
that  were  sufficiently  documented  had  potential  methodological  weaknesses  which  limited  their  conclusiveness  and 
generalization. 


Personnel  performance  was  analyzed  with  two  classes  of  techniques:  performance  models  and  risk  models.  The 
performance  models  showed  no  clear  advantage  of  CBPs  over  PBPs.  Instead,  they  illustrated  the  importance  of 
performance  tradeoffs  in  assessing  different  procedure  systems.  In  general,  complexity  and  attentional  demands 
were  higher,  while  data  retrieval  was  easier  and  task  completion  time  was  less  for  CBPs.  Similarly,  mixed  results 
were  obtained  from  the  risk  analyses.  They  illustrated  the  potential  for  these  systems  to  improve  performance  by 
supporting  such  procedure-related  activities  as  process  monitoring,  logic  analysis,  navigation,  and  place  keeping. 
However,  when  poorly  implemented,  CBPs  can  reduce  human  reliability. 

Finally,  the  SME  review  of  CBPs  identified  many  positive  aspects  of  their  use  on  the  crew’s  performance. 

However,  they  also  identified  a  wide  range  of  issues  to  be  resolved  in  developing  CBPs.  The  review  highlighted 
the  importance  of  considering  HFE  activities  in  CBP  development,  e.g.,  the  integration  of  the  CBP  system  with  the 
other  HSIs  and  with  the  overall  operational  philosophy  of  the  plant.  Thorough  V&V  programs  were  also 
emphasized.  In  general,  these  findings  were  consistent  with  the  information  discussed  earlier. 

When  considering  all  the  results,  we  concluded  that  there  is  evidence  that  CBPs  can  support  and  enhance  operator 
performance.  However,  important  issues  remain  to  be  addressed  both  in  research  and  in  the  development  of 
individual  systems.  Thus,  we  repeat  the  advice  of  researchers  and  developers:  CBP  systems  should  be  developed  in 
such  a  way  that  their  benefits  and  drawbacks  can  be  fully  evaluated  for  each  specific  system.  CBPs  have  important 
impacts  on  NPP  operations,  some  of  which  extend  beyond  those  the  designers  intended. 

Reflecting  this  approach,  we  offer  some  general  considerations  for  near-term  approaches  to  CBP  systems: 

•  Support  cognitive  functions  that  may  be  distracting  and  error  prone,  such  as 

process  monitoring 

logic  analysis  (cautiously  so  not  to  underspecify  the  analysis  and  undermine  operator’s  judgement) 

•  Support  procedure  management,  e.g.,  step  completion,  place  keeping,  transitioning  between  procedures 

•  Provide  PBP  backup  systems  and  ensure  similarity  of  CBPs  and  PBPs  in  order  to  (1)  ensure  confidence  in 
near-term  CBP  applications,  (2)  enable  operating  experience  to  be  gained,  (3)  minimize  the  impact  on 
function  allocation,  (4)  ease  the  training  burdens  associated  with  both  systems,  and  (5)  ensure  successful  crew 
performance  when  transitions  to  and  from  backups  are  necessary  (minimize  the  potential  for  negative  transfer 
or  difficulties  in  performance). 
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HFE  Review  Guidelines 


Guidance  for  the  review  of  CBPs  was  developed  to  address  the  CBP  design  process  and  HFE  design.  Both  types  of 
guidance  are  needed  for  a  design  review.  That  is,  while  there  was  a  sufficient  technical  basis  to  develop  detailed 
guidance  for  design-implementation  review,  as  is  typical  in  NUREG-0700,  several  limitations  in  the  technical  basis 
were  identified.  Many  issues  (listed  below)  remain  for  which  typical  NUREG-0700  guidance  could  not  be 
developed.  Therefore,  until  the  additional  guidance  is  developed,  these  issues  should  be  addressed  for  specific  CBP 
systems  using  CBP  design  process  guidance. 

CBP  Issues 


As  noted  above,  several  human  performance  issues  associated  with  CBPs  were  identified.  They  represent  topics  for 
which  research  is  necessary  before  developing  guidance.  From  a  regulatory  review  perspective,  many  of  them  can 
be  dealt  with  on  a  case-by-case  basis  during  the  design  process  review.  Briefly,  the  issues  included  the  following: 

Methodological  and  Criterion  Requirements  for  Evaluating  CBP  Effects  -  Most  of  the  studies  reviewed  had 
methodological  weaknesses  which  limited  their  conclusiveness  and  generalizability.  This  issue  addresses  the  need 
to  evaluate  CBPs  and  their  effects  on  crew  performance  comprehensively,  to  better  understand  them  under  a  wide 
range  of  scenarios  and  complex  situations,  using  varied  measures  of  personnel  and  system  performance. 

Role  of  Plant  Personnel  in  Procedure  Management  -  This  issue  addresses  the  need  to  determine  how  to  design  and 
review  CBP  systems  (1)  to  allow  operators  to  maintain  an  independent  perspective  and  to  recognize  the 
procedure’s  contribution  to  higher-level  safety  goals,  (2)  to  automate  distracting  and  lower-level  error-prone  tasks, 
and  (3)  to  monitor  the  crew’s  performance,  especially  when  the  crew  and  CBPs  disagree. 

Team  Performance  -  This  issue  addresses  the  requirement  to  explore  the  effect  of  CBPs  on  crew  member’s  roles, 
teamwork,  and  communication.  How  CBPs  can  be  designed  to  effectively  promote  both  is  considered  as  well. 

Situation  Awareness,  Response  Planning,  and  Operator  Error  -  This  issue  addresses  the  need  to  assess  the  effect  of 
CBPs  on  situation  awareness  including: 

•  procedure  management,  such  as  status  of  procedure  steps,  how  procedures  are  structured,  and  the  current 
location  within  a  procedure  or  between  a  set  of  procedures, 

•  the  appropriateness  of  procedures  for  achieving  high-level  procedure  goals,  and 

•  the  plant’s  status. 

Level  of  Automation  of  Procedure  Functions  -  This  issue  addresses  the  need  to  evaluate  the  tradeoffs  between 
automating  procedure  functions,  e.g.,  the  analysis  of  procedure  step  logic,  and  the  operator’s  involvement, 
independence,  and  supervisory  control. 

Keyhole  Effects  and  Use  of  Multiple  CBP  Procedures  -  This  issue  concerns  the  requirement  to  evaluate  the 
significance  of  the  keyhole  effect  in  situations  where  operators  are  required  to  be  in  multiple  procedures  and  must 
access  information  in  parallel. 
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CBP  Failure  in  Complex  Situations  -  This  issue  involves  the  need  to  evaluate  operator’s  management  of  the 
transition  from  CBPs  to  PBPs  and  back  to  CBPs  under  complex  conditions,  e.g.,  in  a  situation  where  operators  are 
deep  into  the  procedures,  multiple  procedures  are  open,  many  steps  are  completed,  many  are  continuously 
applicable,  and  time  and  parameter  steps  are  being  monitored  by  the  CBPs. 

Hybrid  Procedure  Systems  -  This  issue  addresses  the  need  to  evaluate  any  differential  effects  of  having  all  plant 
procedures  presented  in  a  CBP  system  versus  a  hybrid  system,  e.g.,  EOPs  presented  using  CBPs  and  all  other 
procedures  are  paper-based. 

Specific  CBP  Design  Features  -  This  issue  addresses  the  need  to  evaluate  the  relative  effects  of  specific  CBP  design 
features  on  performance. 
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9  GUIDANCE  FOR  CBP  DESIGN  PROCESS  REVIEW 


As  discussed  in  Section  6.1,  the  design  process  guidelines  were  developed  to  address  important  aspects  of  the 
process  and  to  provide  a  means  whereby  human  performance  issues  may  be  assessed  during  a  design  review.  The 
guidelines  were  formatted  to  correspond  to  the  NRCs  general  guidance  for  design  process  review  in  NUREG-071 1 . 
They  are  organized  into  the  following  sections: 

•  HFE  Program  Management 

•  Operating  Experience  Review 

•  Functional  Analysis 

•  Task  Analysis 

•  Staffing 

•  Human  Reliability  Analysis 

•  Human-System  Interface  Design 

•  Procedure  Development 

•  Training  Program  Development 

•  Human  Factors  Verification  and  Validation 

Guidelines  may  specify  that  some  identified  aspect  of  CBPs  needs  to  be  “evaluated.”  NUREG-0700  defines  general 
methods  for  evaluation  and  for  identifying  criteria.  Since  the  guidance  in  this  section  will  eventually  be 
incorporated  into  NUREG-0700,  those  methods  and  criteria  are  not  repeated  below. 

9.1  HFE  Program  Management 

(1)  CBP  design  and  evaluation  should  be  performed  with  a  multidisciplinary  team. 

Discussion:  The  NRC’s  analysis  of  EOPs  (Lapinsky  1989)  noted  that  the  lack  of  a  multidisciplinary  team 
was  associated  with  procedure  deficiencies  which  can  negatively  impact  the  development  and  use  of 
CBPs;  therefore,  a  CBP  development  program  should  address  this  issue.  The  exact  skills  needed  on  the 
multidisciplinary  team  will  vary,  depending  on  the  scope  of  the  CBP  systems.  Appendix  A,  NUREG-071 1 
gives  a  range  of  the  broadest  possible  skills  required;  typical  ones  would  be  those  described  in  Subsections 
2,  4,  6,  7,  9,  and  1 0;  namely.  Systems  Engineering,  I&C  Engineering,  Human  Factors  Engineering,  Plant 
Operations,  Plant  Procedure  Development,  and  Personnel  Training. 

(2)  An  implementation  plan  should  be  developed  to  deal  with  CBP  design,  maintenance,  training  and 
evaluation. 

Discussion:  The  NRC’s  analysis  of  EOPs  (Lapinsky,  1989)  noted  that  the  lack  of  a  systematic  process  was 
associated  with  procedure  deficiencies,  and  can  negatively  impact  the  development  and  use  of  CBPs; 
therefore,  a  CBP  development  program  should  include  this  issue. 
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(3)  The  CBP’s  design  constraints  or  assumptions  should  be  documented  and  their  implications  for  safety 
should  be  evaluated  to  ensure  they  do  not  compromise  the  CBP  system’s  goals. 

Discussion:  Identifying  design  constraints  and  assumptions  is  important  to  HFE  design  in  general  (O’Hara 
et  al.,  1994),  and  for  procedures  systems  in  particular  (Barnes  et  al.,  1996).  Reviewers  should  evaluate  the 
constraints  and  assumptions  applied  by  the  designers  of  the  system  and  specify  their  implications  for 
safety.  These  might  include  limitations  in  the  capabilities  of  the  hardware  or  software,  assumptions  about 
operators’  knowledge  and  skills,  uncertainties  about  particular  content  areas  that  are  not  fully  developed, 
or  decisions  to  leave  the  design  of  some  system  aspects  of  the  CBP  to  other  individuals.  The  reviewer 
should  evaluate  the  acceptability  of  the  designers’  assumptions  and  assure  that  any  constraints  do  not 
compromise  the  system’s  goals. 

(4)  The  CBP  development  program  should  be  fully  documented,  including  design  goals  and  assumptions,  use 
of  operating  exp)erience,  design  analyses,  establishment  of  system  requirements,  tests  and  evaluations, 
detailed  description  of  the  design,  and  verification  and  validation. 

Discussion:  Reviewers  should  be  able  to  follow  the  designers'  development  process  from  the  analysis  of 
requirements  to  the  final  design  and  testing.  In  addition,  reviewers  should  ascertain  that  the  information 
used  to  develop  the  procedures  will  be  retained  in  a  form  accessible  to  the  licensees  who  will  implement 
the  CBPs  at  a  new  or  existing  plant.  Such  records  are  essential  for  assuring  that  the  procedures  can  be 
kept  current  (Barnes  et  al.,  1996;  Lapinsky,  1989). 

9.2  Operating  Experience  Review 

(1)  The  CBP  design  should  eliminate  or  minimize  PBP  problems  where  practical.  Experience  with  paper 
procedures  should  be  reviewed  to  take  advantage  of  lessons  learned  in  their  operational  use,  maintenance, 
and  configuration  control  as  well  as  to  help  ensure  that  problems  in  implementing  PBPs  are  resolved. 

Table  9.1  is  a  partial  list  of  identified  PBP  problems. 

Discussion:  Studies  of  PBPs  at  NPPs,  the  experiences  of  assisting  licensees  in  developing  procedures,  and 
lessons  learned  through  inspections  suggest  that  some  problematic  aspects  of  PBPs  may  be  rectified  by 
computerization  (Barnes  et  al.,  1996). 

(2)  Operating  experience  with  CBP  systems  should  be  reviewed  to  take  advantage  of  lessons  learned  in  using 
the  systems,  as  well  as  to  ensure  that  any  problems  in  implementing  CBPs  are  dealt  with. 

Discussion:  The  review  should  consider  the  use  of  CBPs  in  the  nuclear  industry  and  related  industries. 

(3)  Human  performance  issues,  such  as  visual  fatigue,  arising  from  the  computerization  of  documents  and 
manuals  should  be  addressed. 

Discussion:  Many  HFE  issues  were  identified  that  limited  personnel  performance  when  support  aids  are 
transferred  from  paper  to  computers.  Familiarity  with  this  literature  may  help  to  minimize  these  issues. 

9.3  Functional  Analysis 

An  overall  concept  should  be  developed  of  the  operators*  role  in  managing  and  supervising  plant  procedures. 
Discussion:  Operators  must  be  able  to  supervise  the  conduct  of  procedure  operations,  evaluate  their  success  at 
achieving  safety  goals,  and  formulate  response  plans  when  those  goals  are  not  being  met.  Research  shows  that 
CBPs  can  lessen  operator  independence.  A  clear  statement  of  the  operators*  roles  and  responsibilities  will  help  the 
design  and  training  aspects  of  CBP  development.  The  design  decisions  (e.g.,  scope  and  content,  integration,  and 
function  allocation)  should  flow  clearly  from  the  designers'  overall  concept  of  the  operators'  role  in  managing  plant 
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systems.  Design  documents  should  clearly  articulate  this  concept  and  its  rationale,  and  describe  how  the  design 
carries  out  the  concept  (Barnes  et  al.,  1996). 

9.4  Task  Analysis 

(1)  The  effect  of  the  CBP  on  the  tasks  of  individual  members  of  the  crew  should  be  analyzed,  considering  any 
potential  changes  that  may  result  from  the  combined  use  of  CBPs  and  PBPs,  and  also  the  effect  on 
communications. 


Table  9.1  Examples  of  Deficiencies  in  Paper-Based  Procedures 


Issue 

Deficiency 

Design  Process 

• 

Inadequate  participation  of  operations  and  training  personnel  in  developing 

procedures 

• 

Technically  incorrect  EOPs 

• 

Suitable  under  standard  situations,  but  less  support  in  unusual  situations 

• 

Incomplete  procedures 

• 

inadequate  consideration  of  the  time  required  to  complete  procedural  actions 

• 

Insufficient  verification  and  validation  (V&V)  of  procedures 

Implementation 

• 

Nonspecific  entry  and  exit  conditions  for  support  procedures 

• 

Procedures  are  fixed  and  inflexible 

• 

Incorrect  sequencing  of  action  steps 

• 

Inadequate  consistency  across  procedures 

• 

Inconsistencies  in  formatting  and  use  of  terminology 

• 

Incorrect  identification  of  plant  equipment 

• 

Inadequate  provision  for  varying  level  of  detail 

• 

Non-sequential  presentation  of  information 

• 

Difficulties  in  navigating  to  related  information 

• 

Inadequate  management  of  multiple  procedures 

• 

Unsatisfactory  integration  of  procedure  tasks  and  other  tasks 

• 

Problems  in  labeling  and  headings 

• 

Notes  and  cautions  in  improper  places 

• 

Lack  of  context-dependent  highlighting  and  navigation 

• 

Requirements  to  use  multiple  procedures  simultaneously  and  move  between 

sections 

• 

Lack  of  flowcharts  to  guide  procedure  use 

• 

Inadequate  support  and  reference  material 

• 

Bulkiness 

• 

Physical  handling  of  procedures  near  control  panels 

• 

Separation  from  other  information  sources,  such  as  SPDS 

• 

Inconsistency  with  other  HSIs  in  terms  of  references  to  plant  equipment 

Training 

• 

Operators  poorly  trained  in  using  procedures 
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I  Maintenance  |  •  Maintaining  technical  accuracy  of  procedures  lacking  | 

Discussion:  CBP  may  have  an  important  effect  on  crew  member’s  roles  and  communication.  The  extent 
to  which  they  are  changed  relative  to  PBP  use  may  impact  the  crew’s  situation  awareness  and  plant  safety 
(Roth  and  O’Hara,  1998). 

(2)  CBP  tasks  should  be  analyzed  and  used  as  an  input  to  its  design. 

Discussion:  To  ensure  that  the  design  of  CBPs  is  acceptable  and  usable,  the  results  of  task  analyses 
should  be  incorporated.  Following  the  decisions  on  function  allocation,  the  operators’  tasks  should  be 
defined  at  increasing  levels  of  detail  to  specify  their  actions  and  information  requirements. 

(3)  Tasks  associated  with  CBP  failure  and  back-up  should  be  identified  to  define  the  requirement  for 
indicating  malfunctions.  The  task  of  smoothly  transitioning  from  CBPs  to  a  back-up  method  (such  as 
PBPs)  also  should  be  addressed. 

Discussion:  By  identifying  this  capability  as  a  task,  the  designer  will  then  include  it  in  the  HSl, 
procedures,  and  training.  A  failure  of  the  CBP  System  may  be  total  or  a  more  insidious  partial  one  that  is 
not  obvious  to  the  operators.  Besides  designing-in  indicators  of  failure,  a  means  should  be  identified  to 
smoothly  move  to  the  back-up  system;  the  content  of  the  CBP  and  PBP  (or  other  back-up)  should  be 
compatible. 

9.5  Staffing 

(1)  The  demands  of  operating  and  maintaining  the  CBP  should  be  assessed  for  their  implications  for 
personnel  skills  and  qualifications. 

Discussion:  CBPs  may  impose  demands  on  plant  personnel  that  are  unlike  other  systems,  for  example, 
maintaining  a  large  database.  Human  error  in  that  particular  task  was  identified  as  a  major  cause  of 
events  involving  these  systems  (O'Hzira,  Stubler,  and  Higgins,  1996). 

9.6  Human  Reliability  Analysis 

(1)  Any  effects  on  performance  caused  by  computerization  of  procedures  should  be  analyzed  for  their 
implications  for  those  human  actions  modeled  in  a  PRA. 

Discussion:  PRAs  may  reflect  analyses  of  human  actions  based  on  paper  procedures.  CBPs  have  broad 
effects  on  performance,  both  from  team  and  individual  perspectives.  Accessing  EOPs  through  a  computer 
system  may  create  keyhole  effects  and  may  increase  interface  management  demands.  Some  tasks  may  be 
eliminated,  such  as  monitoring  procedure-specified  parameters,  or  analyzing  procedure  logic.  Also, 
human  errors  in  maintaining  digital  systems  is  a  major  cause  of  events  (O’Hara,  Stubler,  and  Higgins, 
1996).  All  of  these  potential  effects  should  be  considered  in  evaluating  impacts  on  reliability  assumptions 
and  analyses.  Further,  since  operating  experience  with  CBPs  is  limited,  assessing  the  impact  of  CBPs  on 
human  performance  and  reliability  should  utilize,  in  part,  the  results  of  tests  obtained  during  CBP  design, 
evaluation,  verification,  and  validation  (Converse,  1995;  EPRI,  1993a;  Orvis  and  Spurgin,  1996). 

(2)  The  analysis  should  consider  the  effects  on  human  reliability  of  loss  of  CBPs  and  transfer  to  PBPs. 
Discussion:  Using  PBPs  places  different  demands  on  the  crew  and  can  change  their  interaction  and  roles; 
risk  analyses  should  consider  the  implications  of  these  changes. 


NUREG/CR-6634 


9-4 


9  CBP  DESIGN  PROCESS  REVIEW  GUIDANCE 


9.7  Human-System  Interface  Design 

(1)  The  HSl  design  should  consider  methods  by  which  procedure  elements  are  represented  in  the  CBP  and  the 
extent  to  which  usability  principles  for  PBPs  generalize  to  CBP  systems. 

Discussion:  There  are  many  guidelines  for  designing  PBPs.  However,  how  far  they  are  applicable  for 
implementing  a  CBP  system  must  be  assessed.  For  example,  representing  procedure  format,  e.g.,  in  a 
flowchart  or  text,  may  not  reflect  a  simple  application  of  the  PBP  guidance. 

(2)  The  procedure  functions  to  be  provided  by  the  CBP  system  should  be  carefully  analyzed  to  ensure  that  the 
system  is  consistent  with  the  utilities’  general  approach  to  procedure-based  operations,  and  that  the 
operator’s  inputs  and  judgements  are  included,  where  appropriate. 

Discussion:  The  CBP  system  should  provide  operators  with  capabilities  and  functions  to  support  their 
roles  as  system  supervisors  and  their  performance  of  tasks. 

(3)  The  following  aspects  of  CBP  design  should  be  carefully  evaluated  to  ensure  that  the  use  of  procedures  is 
not  jeopardized  and  that  task  requirements  are  adequately  supported: 

•  Number  of  VDUs 

•  Interface  management  and  navigation  functions 

•  Flexibility  of  CBP  display  and  operations 

Discussion:  The  keyhole  effect  that  results  from  the  limited  view  of  plant  information  afforded  by  VDUs, 
interface  management  tasks,  and  computer  system  flexibility  can  significantly  degrade  performance. 

These  aspects  of  the  design  should  be  evaluated  as  part  of  the  design  review. 

(4)  The  potential  interactive  effects  between  procedure  use  and  the  hardware  and  software  used  to  implement 
them  should  be  evaluated. 

Discussion:  NUREG-0700,  Rev.  1  has  guidance  on  hardware  aspects  of  interacting  with  CBPs,  such  as 
VDUs  and  input  devices.  However,  there  may  be  other  such  interactions,  including  those  with  software, 
that  are  not  addressed.  Since  there  is  no  technical  basis  for  guidance,  these  interactions  should  be 
examined  during  the  design  process. 

(5)  The  means  by  which  CBPs  can  support  crew  cooperation,  communication,  and  decision  making  should  be 
evaluated. 

Discussion:  The  NRC  CBP  workshop  and  several  investigations  (e.g.,  Roth  and  O'Hara,  1998) 
highlighted  the  need  to  address  crew  interactions  during  the  design  and  implementation  of  CBPs. 

(6)  Operators  should  be  involved  in  developing  and  evaluating  prototypes  to  ensure  that  their  final  design  is 
usable. 

Discussion:  Lacking  guidance,  CBP  designers  should  have  access  to  system  prototypes,  control  room 
mockups  or  simulations,  and  representatives  of  the  users  to  refine  the  design  of  the  CBPs.  Having  users 
cany  out  procedure  steps  at  the  worksite  (or  a  simulation)  provides  important  information  about  step¬ 
sequencing,  implementation  times,  access  to  the  displays  and  controls,  and  other  physical  characteristics 
of  the  work  environment,  and  an  opportunity  to  collect  their  feedback  on  initial  designs  (Barnes  et  al., 
1996). 
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9.8  Procedure  Development 

9.8.1  Scope  of  Procedures 

(1)  The  purpose  and  scope  of  the  CBP  system  should  be  clearly  defined. 

Discussion:  If  CBPs  encompass  only  some  operator  tasks,  justification  should  be  given  for  excluding 
others.  The  implications  for  operator  performance  of  using  CBPs  and  PBPs  for  different  tasks  should  be 
considered. 

9.8.2  Bases  of  Procedures 

Procedure  bases  refer  to  the  background  information  used  to  develop  the  CBPs.  Procedures  are  critical 
management  tools  because  they  are  among  the  more  important  means  of  guiding  human  interactions  with  the  plant 
systems.  The  procedures  must  not  only  prescribe  technically  correct  actions,  but  must  also  implement  licensee’s 
and  the  NRC’s  expectations  for  the  conduct  of  operations.  Consequently,  their  content  should  be  consistent  with 
the  technical,  regulatory,  and  management  bases  of  plant  operations,  no  matter  what  medium  is  used  to  present 
them. 

(1)  The  technical  bases  for  procedures  shouldTje  documented.  Where  the  documented  bases  for  paper 

procedures  are  unchanged  by  computerization,  the  existing  document  may  be  used.  This  should  include 
the  sources  of  technical  information,  as  well  as  the  process  by  which  the  information  was  used  to  define 
the  desired  operator  actions  and  supplemental  information,  such  as  cautions  and  warnings,  figures,  and 
tables. 

Discussion:  The  technical  bases  for  procedures  are  the  information  used  to  define  the  plant’s  operational 
characteristics  and  may  be  beyond  the  scope  of  a  human  factors  review.  However,  an  HFE  reviewer 
should  evaluate  whether  and  how  the  CBP  designers  used  this  information  to  define  the  operators’  actions 
and  supplemental  information,  such  as  cautions  and  warnings,  figures,  and  tables.  Technical  bases  for 
procedures  should  include  the  following: 

•  Results  of  operational  “lessons  learned” 

•  Technical  guidelines  from  owners’  group 

•  Plant-specific  technical  guidelines 

•  Deviation  documentation 

•  Results  of  safety  analyses  and  accident  analyses 

•  Probabilistic  risk  assessments  (PRAs) 

•  Engineering  documents 

•  Engineering  standards  applied  to  the  design  of  the  plant 

•  Design  criteria  for  the  plant’s  components  and  systems 
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•  Drawings  and  the  specifications  applied  to  designing  and  constructing  the  plant 

•  Records  of  the  basis  for,  and  development  of,  methods  and  calculations 

•  Results  of  design  verification,  qualification  tests  and  functional  tests 

•  Operational  safety  limits  and  technical  specifications 

•  Expected  configuration  of  plant  systems  when  the  procedure  (or  specific  action  step)  is  performed 

•  Other  anticipated  conditions  of  performance 

•  Documentation  of  setpoints 

•  Information  on  equipment  and  component  labeling 

•  Information  on  location  of  equipment  and  components 

(2)  The  regulatory  bases  for  procedures  should  be  specified,  and  the  manner  in  which  they  were  applied  in 
developing  the  CBPs  should  be  documented. 

Discussion:  The  regulatory  bases  for  the  procedures  are  the  requirements  and  guidelines  that  affect, 
constrain  or  are  implemented  by  the  CBPs,  including: 

•  NRC  Rules,  such  as  10  CFR  50.54(m)  pertaining  to  shift-staffing  requirements,  and  10  CFR  50.47(b) 
pertaining  to  emergency  plans 

•  NRC  Regulatory  Guides  and  Standards,  such  as  ANS/ANSI  18.7  (ANS,  1981)  endorsed  in  Reg. 
Guide  1.33,  on  plant  procedures 

•  NRC  guidance  documents,  e.g.,  NUREG-0800  (NRC,  1984);  NUREG-071 1  (O'Hara,  Higgins, 
Stubler,  Goodman,  Eckenrode,  Bongarra,  and  Galletti,  1994);  NUREG-0899  (NRC,  1982),  NUREG- 
1358  and  Supplement  1  to  NUREG-1358  (Galletti  and  Sutthoff,  1992);  generic  communications;  and 
NUREG/CRs 

•  Any  commitments  made  by  the  licensee  to  the  NRC  that  affect  the  procedures 

(3)  The  management  bases  for  procedures  should  be  documented. 

Discussion:  The  management  bases  for  procedures  are  plant  or  site  specific: 

•  The  licensee’s  operational  philosophy 

•  Roles,  responsibilities,  and  authorities  assigned  to  procedure  users 

•  Policies,  programs,  and  plans  for  managing  plant  operations  that  may  affect  the  content  or 
performance  of  the  procedures,  such  as  quality  assurance  or  emergency  response 

•  Requirements  for  adhering  to  procedures 
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•  Requirements  for  independent  verification  of  step  completion  and  accuracy 

(4)  If  the  CBPs  are  to  be  implemented  in  an  operating  plant  using  PBPs,  their  impact  on  existing 
management  bases  should  be  evaluated. 

Discussion:  If  the  CBPs  are  to  be  installed  in  an  existing  plant,  designers  should  obtain  information  on 
the  management  bases  from  plant  personnel.  Introducing  the  CBP  system  may  impact  existing  licensee 
programs  and  procedures,  such  as  operator-licensing  training  programs,  emergency  response  plans,  or  the 
role  of  senior  operations  personnel  in  managing  outages.  The  licensee’s  management,  rather  than  the 
CBP  designers,  should  be  responsible  for  determining  the  impact  of  CBP  design  and  whether  the  changes 
introduced  by  the  CBPs  are  acceptable  or  should  be  revised  to  conform  with  existing  practices.  Those 
policies,  practices,  programs,  and  procedures  affected  should  be  revised  before  implementing  the  system. 

(5)  If  the  CBPs  are  developed  for  a  generic  plzint  design  or  for  new  designs,  plans  and  methods  should  be 
specified  for  incorporating  the  licensee-specific  management  bases.  Since  the  specific  characteristics  of 
the  intended  users  and  their  work  environments  may  not  be  known,  the  methods  by  which  the  CBPs  can 
be  tailored  for  them  should  be  identified. 

Discussion:  For  CBP  designs  for  advanced  control  rooms,  complete  bases  for  the  procedures  may  be 
unavailable.  For  example,  the  plant’s  Technical  Specifications  and  plant-specific  design  information  may 
not  exist  until  an  advanced  reactor  is  built  in  the  United  States  or  a  current  licensee  decides  to  install 
CBPs  in  3n  existing  control  room.  A  final  review  of  the  incorporation  of  the  bases  in  the  CBPs  cannot 
occur  until  plant-  or  site-specific  information  is  available. 

The  management  bases  for  CBPs  for  a  new  plant  will  not  be  available  in  a  generic  design.  Therefore, 
plans  and  methods  for  incorporating  the  management  bases  for  CBPs  should  be  developed  by  their 
designers.  In  addition,  when  CBPs  are  being  developed  for  generic  designs,  the  specific  characteristics  of 
the  intended  users  and  their  work  environments  may  not  be  known;  accordingly,  designers  should  include 
the  following  provisions  for  tailoring  a  CBP  design  for  site-specific  applications: 

•  Any  unique  aspects  of  a  plant's  design  (such  as  characteristics  of  heat  sinks) 

•  The  attributes  of  the  worksite  (e.g.,  ambient  noise  levels,  physical  location  of  required  displays  and 
controls) 

•  The  intended  users  (e.g.,  operator  language,  experience  levels  and  types,  training,  crew  size,  and 
roles) 

9.8.3  Technical  Information 

(1)  The  selection  of  parameters  and  indicators  of  plant  state  to  be  monitored  at  each  procedure  step  should  be 
reviewed. 

Discussion:  How  the  CBPs  use  parameters  and  plant  states  will  affect  the  evaluation  of  procedure  steps 
and  use  by  operators,  as  well  as  the  system’s  design.  Parameters  and  indicators  of  equipment  states 
should  be  appropriate. 

(2)  The  means  by  which  any  the  CBPs  make  the  following  types  of  assessments  should  be  completely 
documented  and  reviewed  by  a  multidisciplinary  team,  including  plant  operators: 

•  Conditions  for  entering  procedures 
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•  Analysis  of  step  logic 

•  Assessment  of  cautions  and  notes 

•  Performance  of  calculations 

•  Assessment  of  exit  conditions  from  procedures 

•  Assessment  of  high-level  procedural  goals 

Discussion:  Undersp)ecifying  procedure  logic  can  cause  misunderstandings  and  potential  errors  in  their 
use.  The  appropriateness  of  the  analyses  must  be  assured,  and  the  role  of  operators’  judgement  accounted 
for.  The  exact  skills  needed  on  the  multidisciplinary  team  will  vary  with  the  scope  of  the  CBP  systems. 
Appendix  A  of  NUREG-071 1  lists  the  range  of  the  broadest  skills  required;  typical  ones  are  those  in 
Subsections  2, 4,  6,  7,  9,  and  10;  namely,  Systems  Engineering,  I&C  Engineering,  Human  Factors 
Engineering,  Plant  Operations,  Plant  Procedure  Development,  and  Personnel  Training. 

(3)  Procedures  should  be  specifically  tailored  to  the  intended  users,  their  physical  work  environment,  and  the 
organization  in  which  the  tasks  are  performed. 

Discussion:  Some  of  the  information  necessary  to  prepare  a  procedure  can  be  developed  generically  by  a 
design  organization  (e.g..  Owners'  Group  Technical  Guidelines  for  EOPs  in  current  plants).  However, 
lessons  learned  in  the  nuclear  power  and  other  industries  showed  that  procedures  must  be  specially 
tailored  to  fulfill  their  functions  of  supporting  users’  accurate  performance,  their  physical  work 
environment,  and  the  organization  in  which  the  tasks  are  performed.  If  there  is  a  mismatch,  procedures 
may  not  be  followed,  or  they  may  be  used  in  unintended  ways.  Because  procedures  also  are  management 
tools,  mismatches  between  licensees'  management  philosophies  and  the  processes  defined  in  procedures 
can  introduce  unintended  organizational  changes  or  break  down  existing  structures  and  processes  (Barnes 
etal.,  1996). 

9.8.4  Maintenance  of  Procedures 

(1 )  Methods  should  be  specified  for  assuring  that  procedure  revisions  do  not  introduce  technical  inaccuracies, 
or  inconsistencies  in  how  the  CBPs  are  presented. 

Discussion:  CBP  designers  should  provide  for  maintaining  the  integrity  of  the  CBPs  and  their  supporting 
documentation.  Because  characteristics  of  users,  systems,  regulatory  requirements,  and  operational  and 
management  practices  change  over  time,  methods  must  be  devised  to  control  revisions  to  the  CBPs  and 
any  documentation  and  databases  on  which  they  depend.  Methods  should  be  specified  for  assuring  that 
revisions  do  not  lead  to  technical  inaccuracies,  or  to  inconsistencies  in  how  the  CBPs  are  presented.  For 
example,  a  CBP  system  may  depend  upon  a  database  to  maintain  a  list  of  required  setpoints  for  different 
conditions  and  automatically  generate  setpoint  information  included  in  procedure  steps.  To  maintain  the 
integrity  of  the  CBPs  if  the  database  is  revised,  it  is  critical  that  the  implications  of  changing  any  value 
can  be  traced  and  controlled  whenever  that  value  appears  in  procedure  steps  (Barnes  et  al.,  1996). 

(2)  Provisions  should  be  made  for  temporarily  changing  procedures.  Administrative  procedures  for 
introducing  and  handling  procedure  changes  should  identify  how  to  properly  implement  the  changes  in 
the  CBP  system.  These  changes  should  be  clearly  identified  in  the  CBP’s  interface. 
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Discussion:  It  should  be  very  clear  to  personnel  what  temporary  changes  were  made  and  whether  an 
aspect  of  the  procedure  being  used  is  a  temporary  one.  Thus,  the  HSl  should  support  such  discriminations 
(Barnes  et  al.,  1996). 

9.9  Training  Program  Development 

( 1 )  The  training  program  should  address  the  role  of  the  operators  to  assure  that  they  remain  in  control  of  the 
CBP  system  and  independently  supervise  it. 

Discussion:  Operators  need  to  understand  the  overall  purpose  of  a  procedure,  and  should  stay  cognitively 
involved  with  its  progress.  They  should  be  trained  to  be  in  control  and  to  question  recommended  steps 
apparently  inconsistent  with  the  overall  procedure  goals  (Blackman  and  Nelson,  1988).  While  operators 
need  to  trust  the  CBP  (Collier,  1996),  overreliance  on  its  information  can  be  a  concern.  For  example, 
while  operators  using  PBPs  monitor  a  variety  of  plant  indications,  operators  using  CBPs  may  not  feel  the 
need  to  look  at  other  sources  of  information  in  the  CR  and,  thus,  may  miss  important  indications  that  are 
not  present  in  the  CBP  (O'Hara,  Stubler,  and  Higgins,  1996). 

(2)  The  knowledge,  skills,  and  abilities  that  users  will  require  to  interact  successfully  with  the  CBP  should  be 
specified  by  the  designers. 

Discussion:  The  demands  of  CBPs  on  personnel  may  be  different  than  those  of  PBPs  (Barnes  et  al., 

1996);  designers  need  to  fully  analyze  personnel  requirements  so  training  can  address  them. 

(3)  The  training  requirements  for  using  CBPs  should  be  specified  and  incorporated  into  a  training  program 
which  should  cover  both  initial  and  ongoing  training.  Training  should  consider  the  design  features, 
functions,  and  limitations  of  CBPs  (such  as  the  potential  for  incorrect  assessments). 

Discussion:  Training  was  identified  as  critical  to  CBP  use  and  may  require  significant  changes  (NRC  CBP 
Workshop). 

(4)  The  training  program  should  inform  op>erators  about  limited  and  complete  failures  of  the  CBP.  Operators 
should  be  trained  to  determine  when  to  override  CBP  evaluations  and  advice.  They  should  be  able  to 
manage  the  transition  to  PBPs  when  CBPs  are  lost  and  move  back  to  them  when  system  function  is 
restored. 

Discussion:  Research  showed  that  operators  may  be  reluctant  to  override  the  CBP’s  advice,  and  may 
believe  the  computerized  procedure  even  when  it  is  wrong  (Blackman  and  Nelson,  1988).  Operators 
should  be  trained  on  making  such  judgements,  and  on  what  to  do  when  they  disagree  with  the  CBP 
(Jeffroy  and  Charron,  1997). 

(5)  The  training  program  should  address  the  importance  of  teamwork  and  communication  when  the  CBP  is 
being  used. 

Discussion:  The  NRC  CBP  Workshop  and  several  investigations  of  CBPs  (e.g.,  Roth  and  O'Hara,  1998) 
correlated  the  importance  of  the  crew’s  communications  and  interactions  to  their  reliability;  this  should  be 
addressed  in  CBP  training. 

(6)  For  CBP  systems  used  for  EOPs  only,  the  compatibility  with  day-to-day  operations  needs  to  be  evaluated 
to  ensure  that  the  system  can  be  easily  understood  and  used. 

Discussion:  IAEA  (1994)  noted  that  the  CBP’s  compatibility  with  day-to-day  operations  needs  to  be 
considered.  Systems  that  have  very  limited  use  in  normal  operations  and  are  only  used  under  infrequent 
special  circumstances  may  have  limited  success.  Thus,  they  noted  that  “...this  may  be  an  issue  for  CBP 
systems  that  are  designed  for  emergency  systems  only”  (p.  31). 
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(7)  The  means  by  which  the  CBP  will  be  introduced  and  implemented  in  an  operating  plant  should  be 
specified. 

Discussion:  Spurgin  et  al.  (1990)  discussed  the  gradual  introduction  of  EOPTS  at  Kuosheng.  It  was  first 
introduced  into  the  training  simulator  (1)  as  an  aid  to  instructors  to  track  the  operators’  responses  to 
accidents,  (2)  as  a  training  tool  for  crews  to  examine  accident-response  strategies,  and  (3)  as  a  tool  to  be 
used  by  crews  in  responding  to  accidents.  During  this  time,  the  correctness  of  EOPTS  was  examined,  and 
several  errors  in  the  PBPs  were  discovered.  Roth  and  O'Hara  (1998)  indicated  the  importance  of  the 
method  of  implementing  the  system  to  the  operators’  subsequent  confidence  in  it. 

9.10  Human  Factors  Verification  and  Validation 

(1)  A  verification  and  validation  (V&V)  plan  should  be  established. 

Discussion:  The  complexity  and  formalization  of  the  plan  will  depend  on  the  scope  of  the  CBP  systems. 

In  some  cases,  it  may  be  part  of  the  overall  CBP  implementation  plan  discussed  in  Section  9.1  (2)  above. 

In  others,  it  should  be  more  formal  and  extensive.  NUREG-0700  and  NUREG-071 1  give  more  detailed 
guidance  on  V&V  plans. 

(2)  V&V  of  procedures  should  ensure  that  the  CBPs  are  technically  correct  and  usable.  Three  types  of  design 
considerations  must  be  addressed  when  evaluating  their  usability:  (1)  HFE  design  standards  and 
guidelines  for  human-computer  interaction,  (2)  HFE  design  guidelines  for  the  format  of  text  instructions 
and  graphics  used  in  presenting  procedural  information,  and  (3)  the  unique  influence  of  site-specific 
characteristics  and  users.  The  acceptability  of  a  CBP  system  cannot  be  determined  without  documentation 
that  all  three  types  of  considerations  were  acceptably  addressed. 

Discussion:  V&V  refers  to  methods  of  ensuring  that  the  CBPs  are  technically  correct  and  usable.  The 
scope,  methods,  timing,  and  composition  of  the  V&V  team  are  important  to  the  success  of  the  system.  For 
CBP  systems  that  select  the  task  instructions  to  be  displayed,  V&V  will  be  a  more  complex  process  than 
for  paper  procedures  or  CBPs  that  are  not  integrated  with  the  plant’s  information  display  and  control 
systems. 

(3)  An  independent  review  team  should  conduct  V&V. 

Discussion:  The  NRC’s  analysis  of  EOPs  (Lapinsky,  1989)  noted  that  the  lack  of  an  independent  review 
to  assure  technical  accuracy  and  usability  was  associated  with  procedure  deficiencies  that  may  negatively 
impact  the  development  and  use  of  CBPs;  therefore,  a  CBP  development  program  should  provide  for 
independent  review.  The  exact  skills  needed  on  the  review  team  will  depend  on  the  scope  of  the  CBP 
systems.  Appendix  A  of  NUREG-071 1  gives  the  broadest  range  of  skills  likely  to  be  required;  typical  ones 
are  those  in  Subsections  2,  4,  6,  7,  9,  and  10  —  namely.  Systems  Engineering,  I&C  Engineering,  Human 
Factors  Engineering,  Plant  Operations,  Plant  Procedure  Development,  and  Personnel  Training. 

(4)  CBP  evaluations  should  use  several  crews  and  scenarios.  They  also  should  use  operator-in-the-loop 
evaluations  to  ensure  that  the  system’s  objectives  are  achieved  and  that  any  transitions  between  CBP  and 
PBP  are  accomplished. 

Discussion:  For  details,  see  Converse,  1995;  EPRI,  1993a;  Orvis  and  Spurgin,  1996. 

(5)  Each  CBP  EOP  procedure  should  be  evaluated  in  the  plant’s  simulator. 

Discussion:  Use  of  simulation  to  evaluate  CBP  systems  was  identified  as  an  important  component  of 
determining  their  acceptability  (EPRI,  1993a). 

(6)  Operators  should  be  able  to  detect  CBP  errors  and  failures. 
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Discussion:  The  NRC  CBP  Workshop  and  several  investigations  of  CBPs  (e.g.,  Roth  and  O'Hara,  1998) 
identified  the  importance  of  crews  being  able  to  detect  errors  and  failures  of  the  CBP  system. 

(7)  The  V&V  should  establish  that  crew  performance  is  not  degraded  as  compared  with  that  reached  using 
PBPs. 

Discussion:  Teamwork  is  essential  to  a  defense-in-depth  approach  to  safely  operating  the  plant  and  any 
failings  can  compromise  it. 

(8)  The  criteria  for  accepting  the  CBP  should  be  specified. 

Discussion:  The  specific  criteria  by  which  the  CBP  will  be  accepted  should  reflect  considerations  of  task- 
performance  criteria,  such  as  task  time  and  error  rate,  determined  by  analysis,  and  of  criteria  based  on  a 
comparison  to  performance  with  PBP  systems. 
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The  guidelines  in  this  section  follow  the  characterization  of  CBP  systems  discussed  in  Section  4.  They  also  reflect 
the  findings  from  our  literature  review  of  the  effects  of  CBPs  on  crew  performance,  specifically  the  identification  of 
functions  discussed  in  Section  5.6.2  as  reflecting  near-term  approaches  to  implementing  CBPs.  According  to  the 
HSI  design  review  procedure  described  in  Part  1  of  NUREG-0700,  Rev.  1,  the  first  step  in  a  design  review  is  to 
select  a  subset  of  guidelines  relevant  to  the  unique  aspects  of  the  particular  design.  There  is  a  wide  range  of  CBP 
designs,  and  some  may  not  include  all  of  the  characteristics  and  functions  in  these  guidelines;  the  reviewer  will 
have  to  determine,  case-by-case,  the  importance  of  CBP  features  that  are  included  in  the  guidelines  but  not  part  of 
the  system  being  reviewed.  This  determination  should  be  based  on  considerations  of  the  specific  purposes  and 
goals  of  that  CBP  system. 

As  described  in  Section  6.2,  guidelines  were  developed  from  the  findings  and  source  materials  reviewed  in 
Section  5.  These  guidelines  were  constructed  in  the  standard  format  adopted  in  NUREG-0700,  Rev.  1  (see 
Section  6.2  of  this  report),  and  organized  into  the  following  sections: 

•  Representation  of  Procedures 

•  Functionality  of  Procedures 

•  Management  and  Support  of  Procedures 

•  CBP  Hardware 

•  CBP  Procedure  Backup 

•  Integration  of  CBPs  with  Other  HSI  Components 

These  new  guidelines  will  be  integrated  into  NUREG-0700,  Rev.  1. 

Guidelines  may  specify  that  some  identified  aspect  of  CBPs  needs  to  be  “evaluated.”  NUREG-0700  defines  general 
approaches  to  methods  of  evaluation  and  for  identifying  criteria.  Since  the  guidance  in  this  section  will  be 
incorporated  into  NUREG-0700,  those  methods  and  criteria  are  not  repeated  below. 

10.1  Representation  of  Procedures 

10.1.1  Identification  of  Procedures 

10.1.1-1  Procedure  Title  and  Identification  Information 

Each  procedure  should  contain  identifying  information  including  title,  procedure  number,  revision 
number,  date,  and  organizational  approval. 

ADDITIONAL  INFORMATION:  This  information  helps  the  user  establish  the  appropriate  context  for 
using  the  procedure. 

Discussion:  This  guideline  was  developed  for  application  to  CBPs  as  an  extension  of  HFE  guidance  for 
paper-based  procedures,  e.g.,  NUREG-0899  (NRC,  1982),  and  from  lessons  learned  discussed  in  Barnes, 
Desmond,  Moore,  and  O'Hara  (1996). 
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10.1.1- 2  High-Level  Goals 

Each  procedure  should  state  its  high-level  goals  and  applicability,  including  its  procedure  category, 
e.g.,  emergency  or  abnormal. 

ADDITIONAL  INFORMATION:  Information  should  be  given  allowing  the  user  to  understand  the 
purpose  or  goal  of  a  series  of  steps  and  supporting  the  user’s  assessment  of  the  success  of  the  procedure  in 
achieving  its  safety  goal. 

Discussion:  Procedure  objectives  need  to  be  emphasized  to  increase  operator’s  awareness  of  the  high-level 
goals  (Bozec  et  al.,  1990;  Wieringa,  Moore,  and  Barnes,  1992).  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principle  of  Situation  Awareness  (see  Appendix  B). 

10.1.2  Basic  Steps 

10.1.2- 1  Concise  Steps 
Procedure  steps  should  be  concise. 

ADDITIONAL  INFORMATION:  Steps  should  be  designed  to  communicate  information  clearly  and 
unambiguously  so  that  they  can  be  easily  understood  and  interpreted  without  error. 

Discussion:  This  guideline  was  developed  for  application  to  CBPs  as  an  extension  of  HFE  guidance  for 
paper-based  procedures,  e.g.,  NUREG-0899  (NRC,  1982),  and  from  lessons  learned  discussed  in  Barnes, 
Desmond,  Moore,  and  O’Hara  (1996).  This  guideline  is  an  application  of  the  High-Level  Design  Review 
Principle  of  Simplicity  of  Design  (see  Appendix  B). 

1 0.1 .2- 2  Short  Sentences 

Procedure  steps  should  be  written  as  short  sentences. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  10.1.2-1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps. 

10.1.2- 3  Active  Voice 

Procedure  steps  should  be  written  in  active  voice. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  10.1.2-1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  1 0.1.2- 1,  Concise  Steps. 

10.1.2- 4  Positive  Commands 

Procedure  steps  should  be  written  as  positive  commands. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  10.1.2-1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps. 

1 0. 1 .2- 5  Sim  pie  Wording 

Short,  simple  words  from  standard  American  English  should  be  used. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  10.1.2-1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps. 

10.1.2- 6  Standard  Punctuation 

Punctuation  should  conform  to  standard  American  English  usage. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  10.1.2-1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps. 
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10.1.2- 7  Consistent  Word  References 

Words,  phrases,  and  equipment  names  and  numbers  should  be  used  consistently  within  and  among 
procedures,  drawings,  other  HSIs,  and  equipment  labels. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  10.1.2-1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10. 1. 2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principle  of  Consistency  (see  Appendix  B). 

10.1.2- 8  Abbreviations  and  Acronyms 

Abbreviations  and  acronyms  should  be  used  consistently  and  limited  to  those  well  known  to  the  users. 
ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  10.1.2-1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10. 1.2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principles  of  User  Model  Compatibility  and  Consistency  (see  Appendix  B). 

10.1.2- 9  Units  of  Measures 

Numerical  information  should  include  units  of  measure. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  10.1.2-1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps. 

10.1.2- 10  Numerical  Precision 

Numbers  should  be  specified  at  the  appropriate  precision. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  1 0.1.2- 1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10.1 .2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principle  of  Task  Compatibility  (see  Appendix  B). 

10.1.2- 11  Number  Ranges 

Ranges  of  numbers  should  be  specified,  rather  than  error  bands. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  10. 1. 2-1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principles  of  Situation  Awareness,  Task  Compatibility,  and  Cognitive 
Workload  (see  Appendix  B). 

10.1.2- 12  Use  Arabic  Numerals 
Arabic  numerals  should  be  used. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  10.1.2-1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10. 1. 2- 1,  Concise  Steps. 

10.1.2- 13  Spelled  Numbers 

Numbers  that  are  spelled  out  should  be  consistently  spelled  under  the  same  conditions. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  I0.1.2-I,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps. 

10.1.2- 14  Presentation  of  Conditional  Steps 

Conditional  steps  should  be  shown  in  traditional  text  formats  following  the  guidance  in  Appendix  B  of 
NUREG-0899. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  1 0.1. 2- 1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps. 


10-3 


NUREG/CR-6634 


10  CBP  HFE  DESIGN  REVIEW  GUIDELINES 


10.1.2- 15  Specification  of  Preconditions  for  Steps 

The  procedure  should  specify  any  conditions  that  must  be  met  before  an  action  can  be  undertaken. 
ADDITIONAL  INFORMATION:  Information  about  preconditions  in  the  procedure  should  be  located  so 
that  users  read  the  information  before  acting.  Information  given  in  other  locations  may  be  overlooked,  or 
require  additional  actions  to  retrieve  it,  which  may  be  distracting  and  time  consuming.  Further,  if 
conditions  are  implied,  users  may  easily  miss  or  misinterpret  them. 

Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principles  of  Situation  Awareness,  Task  Compatibility,  Timeliness,  Feedback, 
and  Response  Workload  (see  Appendix  B). 

10.1.3  Warnings,  Cautions,  Notes,  and  Supplementary  Information 

10.1.3- 1  Parallel  Display  with  Procedure  Step 

The  warnings  and  cautions  applicable  to  a  single  step  (or  to  a  series  of  steps)  should  be  displayed  when 
the  step(s)  is  on  the  screen. 

ADDITIONAL  INFORMATION:  Displaying  warnings  and  cautions  at  the  same  time  as  their  associated 
procedure  steps  will  help  ensure  that  users  read  the  information  when  they  evaluate  the  step.  Information 
provided  elsewhere  may  be  overlooked,  or  may  require  retrieval  by  distracting  and  time-consuming 
actions. 

Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principles  of  Situation  Awareness,  Timeliness,  and  Response  Workload  (see 
Appendix  B). 

10.1.3- 2  Position  Before  Action  Steps 

Warnings,  cautions,  and  notes  should  be  presented  so  that  they  will  be  read  before  the  applicable  action 
steps. 

ADDITIONAL  INFORMATION:  Displaying  warnings,  cautions,  and  notes  before  action  steps  will  help 
ensure  that  users  will  read  the  information  before  taking  action.  Information  provided  in  other  places  may 
be  overlooked  or  may  be  distracting  and  time  consuming  to  retrieve. 

Discussion:  See  discussion  for  Guideline  10.1 .2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principles  of  Situation  Awareness,  Task  Compatibility,  Feedback,  and 
Timeliness  (see  Appendix  B). 

10.1.3- 3  Action  References 

Warnings,  cautions,  and  notes  should  not  include  implied  or  actual  action  steps. 

ADDITIONAL  FNFORMATION:  Actions  should  be  specified  in  procedure  steps  only. 

Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principle  of  Task  Compatibility  (see  Appendix  B). 

10.1.3- 4  Distinction  from  Other  Procedure  Elements 

Warnings,  cautions,  and  notes  should  be  uniquely  presented,  so  that  they  are  easily  distinguished  from 
each  other  and  from  other  display  elements. 

Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principles  of  Task  Compatibility  and  Organization  of  HSI  Elements  (see 
Appendix  B). 
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10.1.3- 5  Supplementary  Information 

All  supplementary  information  (such  as  tables  and  figures)  required  for  a  procedure  step  and  available  to 
the  CBP  should  be  shown  on  the  screen  concurrently  with  the  step,  or  on  another  easily  viewed  display. 
Discussion:  See  discussion  for  Guideline  10. 1. 2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principles  of  Situation  Awareness,  Task  Compatibility,  and  Response 
Workload  (see  Appendix  B). 

10.1.4  Lists 

10.1.4*1  Appropriate  Application  of  Lists 

Groups  of  three  or  more  related  items  (e.g.,  actions,  conditions,  components,  criteria,  systems)  should  be 
presented  as  a  list. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  10.1.2-1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principle  of  Organization  of  HSl  Elements  (see  Appendix  B). 

10.1.4- 2  Distinction  from  Other  Procedure  Elements 

Formatting  should  be  used  to  differentiate  items  in  a  list  from  other  procedure  elements. 

ADDITIONAL  INFORMATION:  See  additional  information  in  Guideline  10.1.2-1,  Concise  Steps. 
Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps. 

10.1.4- 3  Identification  of  Precedence 

The  presence  or  absence  of  precedence  among  items  in  lists  should  be  indicated. 

ADDITIONAL  INFORMATION:  It  should  be  clear  to  users  whether  some  items  take  precedence  over 
others. 

Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principle  of  Situation  Awareness  (see  Appendix  B). 

10.1.4- 4  List  Overviews 
Overviews  should  introduce  each  list. 

ADDITIONAL  INFORMATION:  An  example  of  an  overview  is  “Ensure  that  all  of  the  following  tests 
were  completed:” 

Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principle  of  Situation  Awareness  (see  Appendix  B). 

10.1.4- 5  Assuring  Users’  Attention 

The  method  for  assuring  that  each  item  in  a  list  has  received  the  users’  attention  should  be  consistent. 
ADDITIONAL  INFORMATION:  For  example,  an  electronic  checklist  may  be  provided  so  that  operators 
can  check  off  items  they  have  attended  to.  If  operators  proceed  before  all  items  are  checked  off,  the  CBP 
may  alert  them  to  the  unchecked  items. 

Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principle  of  Task  Compatibility  (see  Appendix  B). 

10.1.5  Organization  of  Procedures 

10.1.5- 1  Hierarchical,  Logical  Organization 

The  procedures  should  be  organized  in  a  hierarchical,  logical,  consistent  manner. 
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ADDITIONAL  INFORMATION:  Organization  will  make  it  easier  for  users  to  see  the  relationships 
among  procedures. 

Discussion:  See  discussion  for  Guideline  10.1.2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principles  of  Logical/Explicit  Structure  and  Consistency  (see  Appendix  B). 

10.1.5- 2  Organization  of  Procedure  Steps 

Each  procedure  should  be  organized  into  sections  of  related  steps. 

Discussion:  See  discussion  for  Guideline  10. 1. 2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principles  of  Logical/Explicit  Structure  and  Consistency  (see  Appendix  B). 

10.1.6  Formatting  and  Screen  Layout 

10.1.6- 1  Organization  Format  of  Procedures 
The  procedure’s  format  should  reflect  its  organization. 

ADDITIONAL  INFORMATION:  Formatting  methods  to  indicate  the  organization  of  a  procedure  may 
include  the  use  of  headings  or  colors  to  distinguish  parts  of  the  procedure. 

Discussion:  See  discussion  for  Guideline  10. 1. 2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principles  of  Logical/Explicit  Structure  and  Consistency  (see  Appendix  B). 

10.1.6- 2  Format  of  Procedures 

A  consistent  format  should  be  used  to  display  procedures. 

ADDITIONAL  INFORMATION:  Whether  procedures  are  presented  in  text,  flowchart,  or  otherwise,  a 
consistent  approach  across  procedures  will  facilitate  using  and  moving  between  multiple  procedures. 
Discussion:  There  is  insufficient  research  to  specify  one  format  over  another  for  presenting  CBPs. 

Further,  it  is  important  that  CBPs  are  consistent  with  paper  procedures.  However,  whatever  format  is 
used,  consistency  supports  the  rapid  use  of  information  when  moving  within  and  between  procedures,  and 
enables  operators  to  form  expectancies  which  can  reduce  the  workload  of  finding  information.  This  will 
also  speed  procedure  use  and  reduce  errors.  This  guideline  is  an  application  of  the  High-Level  Design 
Review  Principles  of  Logical/Explicit  Structure  and  Consistency  (see  Appendix  B). 

10.1.6- 3  Partitioning  Procedures 

A  consistent  approach  to  partitioning  procedures  should  be  used. 

ADDITIONAL  INFORMATION:  Partitioning  refers  to  how  a  procedure  is  organized  to  be  displayed  on 
the  VDU  screen.  For  example,  it  may  be  divided  into  distinct  pages,  and  users  would  navigate  from  one 
to  the  next.  Alternatively,  it  may  be  presented  as  one  continuous  display  that  the  user  scrolls. 

Discussion:  Unlike  PBPs,  CBPs  are  viewed  through  the  limited  display  area  of  one  or  more  VDUs.  Thus, 
regardless  of  format,  the  designer  must  decide  whether  the  procedure  will  appear  as  a  continuous 
scrollable  display  or  be  divided  into  discrete  pages.  This  guideline  is  an  application  of  the  High-Level 
Design  Review  Principles  of  Task  Compatibility,  Logical/Explicit  Structure,  and  Consistency  (see 
Appendix  B). 

1 0. 1 .6- 4  Organization  of  Display  Screen 

Each  display  screen  should  locate  information  and  HSI  features  consistently. 

ADDITIONAL  INFORMATION:  When  the  information  and  features,  such  as  procedure  steps,  controls, 
and  navigation  aids  are  consistently  located,  users’  performance  improves  because  expectations  can  guide 
the  search  for  information,  and  reduce  the  time  and  workload  associated  with  finding  it. 

Discussion:  See  discussion  for  Guideline  10. 1. 2- 1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principles  of  Logical/Explicit  Structure  and  Consistency  (see  Appendix  B). 
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10.1.6- 5  Continuously  Presented  Procedure  Information 

The  procedure’s  title  and  identification  should  be  continuously  presented. 

ADDITIONAL  INFORMATION:  This  information  helps  set  the  context  for  the  overall  procedure  within 
which  its  steps  are  interpreted.  It  is  especially  important  when  more  than  one  procedure  can  be  open  at 
one  time. 

Discussion:  One  concern  identified  with  CBP  systems  is  the  loss  of  awareness  of  the  context  in  which 
procedures  are  used,  e.g.,  high-level  safety  goals  and  plant  status  (NRC  CBP  Workshop;  Roth  and  O'Hara, 
1998;  Spurgin  et  ah,  1990).  The  identifying  information  maintains  focus  on  the  way  in  which  individual 
steps  are  interpreted,  especially  when  multiple  procedures  are  in  use.  This  guideline  is  an  application  of 
the  High-Level  Design  Review  Principle  of  Situation  Awareness  (see  Appendix  B). 

10.1.6- 6  Continuously  Presented  Status  of  High-Level  Goals 

The  status  of  high-level  procedure  goals  should  be  continuously  presented. 

ADDITIONAL  INFORMATION:  This  information  helps  set  the  overall  context  in  which  procedure  steps 
are  interpreted.  Continuous  presentation  of  high-level  goal  status,  such  as  status  of  critical  safety 
functions,  will  facilitate  users'  awareness  of  them,  particularly  when  more  than  one  procedure  is  open 
simultaneously. 

Discussion:  The  loss  of  awareness  of  the  context  in  which  procedures  are  used,  e.g.,  high-level  safety 
goals  and  plant  status  is  a  concern  with  CBPs  (NRC  CBP  Workshop;  Roth  and  O'Hara,  1998;  Spurgin,  et 
al.  1990).  Awareness  of  high-level  goals  is  important  to  interpreting  individual  steps  and  for  determining 
which  procedure  is  appropriate.  Roth  and  O'Hara  (1998)  also  observed  that  the  most  significant  series  of 
information  needed  by  operators  on  loss  of  the  CBP  was  critical  safety  function  status.  This  guideline  is 
an  application  of  the  High-Level  Design  Review  Principles  of  Situation  Awareness  and  Timeliness  (see 
Appendix  B). 

10.2  Functionality  of  Procedures 
10.2.1  Supervision  and  Control  of  Procedures 

10.2.1- 1  Users’  Control  of  Procedure  Path 

Users  should  be  in  control  of  the  sequence  of  steps  that  are  followed. 

ADDITIONAL  INFORMATION:  Most  procedures  have  specifically  defined  steps  that  have  to  be 
performed  sequentially,  and  others  that  can  be  varied  at  the  operator's  discretion;  CBPs  should  identify 
which  one  is  applicable.  However,  operators  should  have  the  flexibility  to  move  around  within  the 
procedure,  so  that  they  can  check  and  make  verifications. 

Discussion:  The  CBP  guidance  level  should  leave  the  operators  in  the  loop  so  they  retain  control  and  are 
the  final  authority  (EPRI,  1993a;  Dien  et  al.,  1991).  This  guideline  is  an  application  of  the  High-Level 
Design  Review  Principles  of  Cognitive  Compatibility  and  Situation  Awareness  (see  Appendix  B). 

10.2.1- 2  Users’  Control  of  Pace  of  Procedures 

Users  should  be  in  control  of  the  pace  at  which  procedure  steps  are  followed. 

ADDITIONAL  INFORMATION:  Operators  need  to  maintain  situation  awareness  of  procedure-related 
decisions.  To  accomplish  this,  they  must  be  in  control  of  the  pace  at  which  steps  are  followed. 

Discussion:  The  operator  should  retain  control  and  should  be  the  final  authority  (EPRI,  1993a;  Dien  et 
al.,  1991).  This  guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Cognitive 
Compatibility  and  Situation  Awareness  (see  Appendix  B). 
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10.2.1- 3  Understandability  of  Analysis  of  Procedure  Steps 

The  methods  by  which  CBPs  analyze  procedure  steps  should  be  consistent  with  the  methods  by  which 
users  analyze  steps  in  procedure  logic  steps,  so  that  the  results  are  understandable. 

ADDITIONAL  INFORMATION:  Users  must  be  able  to  judge  the  acceptability  of  the  CBP’s  advice  and 
recommendations. 

Discussion:  To  maintain  their  role  of  system  supervisors,  operators  need  to  be  able  to  understand  and 
evaluate  the  appropriateness  of  procedure  analyses.  The  CBP  should  not  require  the  operator  to  conform 
to  its  method  of  analysis  (Bernard,  1989).  This  guideline  is  an  application  of  the  High-Level  Design 
Review  Principles  of  Task  Compatibility  and  Situation  Awareness  (see  Appendix  B). 

10.2.1- 4  Users’  Verification  of  CBP  Information 

The  users  should  be  able  to  verify  the  system's  assessment  of  plant  status. 

ADDITIONAL  INFORMATION:  This  verification  includes  process  parameters,  equipment  status, 
analysis  of  procedure  step  logic,  and  evaluation  of  cautions.  Any  analysis  done  by  the  CBP  should  be 
accessible  to  users  for  review. 

Discussion:  To  maintain  their  role  of  system  supervisors,  operators  need  to  be  able  to  access  information 
enabling  them  to  determine  the  appropriateness  of  procedure  information.  This  guideline  is  an 
application  of  the  High-Level  Design  Review  Principles  of  Task  Compatibility  and  Situation  Awareness 
(see  Appendix  B). 

10.2.1- 5  Users’  Override  of  CBP 

Users  should  be  able  to  override  any  CBP  information,  calculation,  evaluation,  or  assessment. 

Discussion:  Operators  should  be  able  to  override  a  course  of  action  suggested  or  recommended  by  a  CBP 
system.  This  is  necessary  for  situations  in  which  the  operator  has  access  to  information  that  is  not 
available  to  the  CBP,  the  CBPs  guidance  is  too  strict,  or  when  the  CBP  uses  out-of-date  in  formation 
(Bozec  et  al.,  1990).  This  guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of 
Cognitive  Compatibility  and  Situation  Awareness  (see  Appendix  B). 

10.2.2  Monitoring  and  Assessment  of  Procedures 

10.2.2- 1  Automatic  Identification  of  Procedures 

The  CBP  should  alert  users  when  entry  conditions  to  a  procedure  are  satisfied. 

ADDITIONAL  INFORMATION:  This  capability  will  help  users  determine  the  appropriate  procedures  for 
the  existing  plant  situation. 

Discussion:  This  capability  was  identified  as  being  beneficial  to  crew  reliability  (Orvis  and  Spurgin, 

1996).  This  guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Situation 
Awareness  and  Cognitive  Workload  (see  Appendix  B). 

10.2.2- 2  Automatic  Monitoring  of  Plant  Parameters  and  Equipment  Status 

The  CBP  should  automatically  provide  accurate  and  valid  information  on  the  values  of  parameters  and 
status  of  equipment,  when  they  are  available  to  the  system. 

ADDITIONAL  INFORMATION:  It  should  be  clear  to  operators  what  specific  information  is  used  as  the 
source  of  these  actual  values  and  states. 

Discussion:  Supporting  cognitive  functions,  such  as  obtaining  parameter  values  (monitoring)  may  reduce 
the  demands  on  attentional  resources  and  working  memory  and  enable  the  operator  to  focus  more  on 
evaluating  higher-level  procedure  goals.  It  may  also  help  solve  PBP  issues.  This  capability  was  identified 
as  being  beneficial  to  the  crew’s  reliability  (Orvis  and  Spurgin,  1996;  Pirus  and  Chambon,  1997;  Niwa  et 
al.,  1996).  Further,  presenting  plant  parameters  and  status  in  procedure  steps  is  a  URD  requirement 
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(EPRI,  1993a).  This  guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Situation 
Awareness  and  Cognitive  Workload  (see  Appendix  B). 

10.2.2- 3  Frequent  Monitoring 

The  CBP  should  frequently  monitor  procedure-defined  parameters. 

ADDITIONAL  INFORMATION:  Frequent  monitoring,  such  as  twice  a  second,  promptly  notifies  users  of 
status  changes. 

Discussion:  The  continuous  updating  of  plant  parameters  and  status  is  identified  as  a  URD  requirement 
(EPRI,  1993a).  This  guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Situation 
Awareness,  Cognitive  Workload,  and  Timeliness  (see  Appendix  B). 

10.2.2- 4  Automatic  Calculation  of  Procedure-Referenced  Values 

The  system  should  undertake  calculations,  such  as  subcooling  margin,  that  are  required  when  using 
procedures. 

Discussion:  The  capability  to  perform  calculations  was  identified  as  an  important  feature  of  CBPs  (Roth 
and  O’Hara,  1998;  Barnes  et  al.,  1996).  This  guideline  is  an  application  of  the  High-Level  Design  Review 
Principle  of  Cognitive  Workload  (see  Appendix  B). 

10.2.2- 5  Analysis  of  Step  Logic 

The  CBP  should  evaluate  the  logic  of  each  procedure  step  and  show  the  results  to  the  user. 

ADDITIONAL  INFORMATION:  Procedure  steps  often  contain  logical  relationships;  for  example, 
actions  are  to  be  performed  if  an  identified  set  of  conditions  exists.  The  analysis  of  these  logical 
relationships  must  be  carefully  verified  to  avoid  underspecification.  This  occurs  when  the  logic  used  to 
resolve  a  procedure  step  is  too  simplified,  and  does  not  address  all  of  the  considerations  that  operators  do 
when  evaluating  the  step. 

Discussion:  Supporting  cognitive  functions,  such  as  comparing  actual  parameter  values  to  reference 
values  (resolution  of  procedure  step  logic)  may  reduce  the  demands  on  attentional  resources  md  working 
memory,  and  enable  the  operator  to  focus  on  evaluating  high-level  procedure  goals.  This  CBP  capability 
was  identified  as  a  major  benefit  and  one  which  helped  operators  to  follow  the  procedures  correctly,  and  to 
interpret  the  logical  statements  that  are  a  part  of  the  procedure  steps  (Spurgin  et  al.,  1990).  It  also  was 
thought  to  improve  the  crew’s  reliability  (Orvis  and  Spurgin,  1996;  Moieni  and  Spurgin,  1993b).  This 
guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Situation  Awareness  and 
Cognitive  Workload  (see  Appendix  B).  However,  while  this  is  a  potentially  powerful  feature,  it  must  be 
used  cautiously.  Some  procedural  details  are  intentionally  left  relatively  abstract  because  they  require  the 
operator’s  judgement  on  the  basis  of  local  knowledge;  e.g.,  knowledge  of  equipment  availability  and  status 
at  the  current  time.  While  computerization  can  increase  the  detail,  this  could  inadvertently  change  the 
procedure’s  context  and  the  operator's  interpretation  of  it.  Thus,  underspecification  of  the  logic  can  be  an 
issue  (O'Hara,  Stubler,  and  Higgins,  1996;  Roth  and  O’Hara,  1998).  Further,  the  CBP  is  not  fully  aware 
of  what  operators  are  doing  nor  of  their  intentions  (Blackman  and  Nelson,  1 988). 

10.2.2- 6  Continuous  Analysis  of  Non-Current  Step  Logic 

Steps  of  continuous  applicability,  time-dependent  steps,  and  process-dependent  steps  should  be  monitored 
by  the  CBP  and  the  user  should  be  alerted  when  conditions  in  those  steps  become  effective. 

ADDITIONAL  INFORMATION:  The  analysis  must  be  carefully  verified  to  avoid  underspecifying  its 
logic.  The  alert  should  not  automatically  remove  the  user’s  current  display.  Instead,  it  should  be 
presented  as  a  supplemental  display  or  as  an  alert. 

Discussion:  See  discussion  of  the  previous  guideline.  In  addition,  operators  prefer  that  procedures  not 
automatically  reset  or  return  to  a  previous  step  when  there  is  a  change  in  process  status;  instead  automatic 
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monitoring  of  previous  steps  and  indications  of  a  change  in  their  status  is  preferred  (Bozec  et  al.,  1990). 
This  guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Situation  Awareness  and 
Cognitive  Workload  (see  Appendix  B). 

10.2.2- 7  Coding  of  Logical  Analysis 

When  procedure’s  step  logic  indicates  a  violation  of  the  step,  the  information  should  be  coded  to  make 
that  step  more  salient  to  users. 

Discussion:  Handling  of  multiple  procedures  is  easier  when  the  relevant  information  in  each  is 
highlighted.  When  operators  transition  from  one  to  another,  the  highlighted  information  directs  them  to 
the  appropriate  location  (Pirus  and  Chambon,  1997).  This  guideline  is  an  application  of  the  High-Level 
Design  Review  Principle  of  Situation  Awareness  (see  Appendix  B). 

10.2.2- 8  Analysis  of  Cautions 

The  conditions  described  in  cautions  should  be  automatically  monitored  by  the  CBP  system,  and  the  user 
should  be  alerted  when  the  caution  is  in  effect. 

ADDITIONAL  INFORMATION:  Evaluating  cautions  and  alerting  users  to  their  applicability  will  ensure 
that  users  will  read  the  information  at  the  appropriate  time,  and  reduce  the  chance  that  it  may  be 
overlooked.  The  conditions  for  cautions  must  be  established  with  care  such  that  the  logic  is  not 
underspecified. 

Discussion:  Supporting  cognitive  functions,  such  as  monitoring  caution  conditions  and  comparing  their 
reference  values,  may  reduce  the  demands  on  attentional  resources  and  working  memory  and  enable  the 
operator  to  attend  more  to  the  higher-level  procedure  goals.  Alerting  operators  to  applicable  cautions  will 
help  ensure  that  they  are  not  overlooked.  This  guideline  is  an  application  of  the  High-Level  Design 
Review  Principles  of  Situation  Awareness  and  Cognitive  Workload  (see  Appendix  B). 

10.2.2- 9  Coding  Applicable  Cautions 

CBPs  should  use  coding  to  indicate  when  a  caution  is  in  effect. 

ADDITIONAL  INFORMATION:  Coding  techniques,  such  as  color  coding,  may  be  used  to  enhance  the 
salience  of  important  information. 

Discussion:  This  guideline  is  an  application  of  the  High-Level  Design  Review  Principle  of  Situation 
Awareness  (see  Appendix  B). 

10.2.2- 10  Users’  Acknowledgment  of  Procedure  Analyses 

Users  should  make  some  form  of  acknowledgment  of  procedure  steps  and  recommendations  for 
terminations  and  transitions. 

ADDITIONAL  INFORMATION:  As  an  example,  operators  may  acknowledge  that  a  step  is  satisfied  by 
depressing  the  “Return”  key,  or  clicking  on  an  onscreen  acceptance  button.  Such  acknowledgment  helps 
the  operators  to  maintain  awareness  of  the  procedure’s  status. 

Discussion:  The  CBP  guidance  level  should  leave  the  operator  in  the  loop,  so  they  retain  control  and  are 
the  final  authority  (EPRI,  1993a;  Dien  et  al.,  1991).  This  guideline  is  an  application  of  the  High-Level 
Design  Review  Principles  of  Cognitive  Compatibility  and  Situation  Awareness  (see  Appendix  B). 

10.2.2- 11  Identification  of  User  Input  Requirements 

The  CBP  should  provide  users  with  clear,  timely  indications  when  they  need  to  input  any  information  not 
available  to  it. 

ADDITIONAL  INFORMATION:  CBPs  may  rely  on  users  to  for  process  parameter  values,  equipment 
status  (such  as  whether  a  valve  is  open  or  closed),  analyses  of  logic  steps  where  users’  judgement  is 
involved,  or  to  assess  any  conditions  not  within  the  capability  of  the  CBP. 
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Discussion:  While  the  CBP  monitors  the  system  through  the  l&C,  operators  must  provide  some 
information.  Failures  to  do  so  can  lead  to  incorrect  assessments  and  guidance  from  the  CBP  (Jeffroy  and 
Charron,  1997).  This  guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of 
Logical/Explicit  Structure  and  Timeliness  (see  Appendix  B). 

10.2.2- 12  Adjustable  Level  of  Detail 

Users  should  be  able  to  choose  the  level  of  detail  with  which  procedures  are  presented. 

ADDITIONAL  INFORMATION:  While  plant  practices  on  using  procedures  may  be  specified  by 
management,  there  may  be  flexibility  in  the  level  of  detail  that  can  be  provided.  For  example,  users  may 
want  less  detail  when  a  procedure  step  is  satisfied.  Alternatively,  a  user  may  choose  to  see  all  of  the 
individual  evaluations  leading  to  the  conclusion  that  the  step  was  satisfied.  This  must  be  done  with  care 
so  that  it  does  not  affect  the  interpretation  of  procedure  information.  Also,  users  should  be  trained  as  to 
how  and  when  to  vary  levels  of  detail. 

Discussion:  Procedural  guidance  can  be  used  more  efficiently  when  CBPs  can  adjust  the  level  of  detail  for 
operators  with  varying  familiarity  with  the  tasks,  components,  systems,  and  processes  defined  in  the 
procedures.  This  may  also  help  address  a  deficiency  of  PBPs.  It  was  identified  as  a  desirable  feature  of 
CBPs  by  many  studies  (NRC  CBP  Workshop;  Dien  et  al.,  1991).  However,  providing  too  much  detail 
should  be  avoided  (Bozec  et  al.,  1990;  Roth  and  O'Hara,  1998),  especially  for  exp)erienced  operators  (Niwa 
et  al.,  1996).  This  guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Flexibility 
(see  Appendix  B). 

1 0.2.2- 13  Context-Specific  Guidance 

Procedure  guidance  should  be  context  sensitive  where  possible. 

ADDITIONAL  INFORMATION:  For  example,  the  CBP  system  should  not  indicate  an  action  to  start  a 
pump  when  it  can  determine  that  the  pump  is  already  running. 

Discussion:  A  general  problem  observed  with  COSSs  is  that  the  information  is  “aeon textual,”  i.e.,  their 
guidance  had  little  reference  to  the  current  situation  (Dien  and  Montmayeul,  1995).  For  CBPs,  this 
problem  can  be  corrected  by  supporting  procedure  sensitivity  to  the  current  situation  (Niwa  et  al.,  1996). 
Removing  information  inappropriate  to  the  current  situation  and  which  is,  therefore,  potentially 
distracting  and  uses  up  valuable  time,  will  help  operators  to  concentrate  on  important  information.  This 
guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Cognitive  Compatibility  and 
Situation  Awareness  (see  Appendix  B). 

1 0.2.2- 14  Assessment  of  High-Level  Goal  Status 

The  CBP  should  continuously  assess  and  present  the  status  of  higher-level  safety  goals,  such  as  critical 
safety  functions,  and  alert  the  user  to  any  challenges. 

Discussion:  Supix)rting  cognitive  functions,  such  as  comparing  parameter  values  to  goal-reference  values, 
may  reduce  the  demands  on  attentional  resources  and  working  memory  and  enable  the  operator  to  better 
attend  to  determining  the  success  of  the  procedure  in  achieving  the  higher-level  goals.  Alerting  operators 
to  possible  challenges  will  help  ensure  that  they  will  not  be  overlooked.  The  availability  of  safety-goal 
status  is  important  to  operators’  overall  assessment  of  the  procedure  (Roth  and  O'Hara,  1998).  This 
guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Situation  Awareness  and 
Cognitive  Workload  (see  Appendix  B). 

10.2.2- I5  Assessment  of  Conditions  Terminating  a  Procedure 

The  CBP  should  automatically  identify  when  conditions  are  met  for  transitioning  or  exiting  from  a 
procedure. 
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ADDITIONAL  INFORMATION:  This  capability  will  help  users  determine  when  procedures  they  are 
using  are  no  longer  appropriate  for  the  existing  situation. 

Discussion:  By  helping  users  determine  when  procedures  become  inappropriate  for  the  existing  situation, 
the  chances  of  operators  delaying  actions  identified  in  the  appropriate  procedure  are  reduced.  This 
guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Situation  Awareness  and 
Cognitive  Woricload  (see  Appendix  B). 

10.2.3  Monitoring  Users’  Actions 

10.2.3- 1  Monitoring  Users 

User  responses  to  procedures  should  be  monitored  and  recorded  by  the  CBP. 

ADDITIONAL  INFORMATION:  Monitoring  information  on  users’  input  to  information  requested  by  the 
procedure  and  their  subsequent  actions  is  necessary  if  the  CBP  is  to  properly  assess  appropriate  procedural 
pathways. 

Discussion:  CBPs  should  be  designed  to  maintain  information  on  what  the  crew  is  doing  that  is  relevant 
to  implementing  the  procedure  (Blackman  and  Nelson,  1988).  To  evaluate  procedure  steps  the  operators 
must  be  aware  of  the  information  being  analyzed  by  the  CBP.  To  the  extent  that  the  CBP  system  has 
information  on  users’  actions,  it  can  perform  this  task  more  effectively. 

10.2.3- 2  Alert  Users  to  Deviations  in  Procedure 

Users  should  be  alerted  if  their  input  is  incorrect,  or  when  their  actions  are  not  consistent  with  CBP 
evaluations. 

ADDITIONAL  INFORMATION:  The  alert  should  be  advisory  and  not  discourage  the  user’s  actions. 

This  feature  must  be  supported  with  training,  so  users  are  not  reluctant  to  go  against  the  CBP’s 
evaluations. 

Discussion:  EPRJ  suggested  that  CBPs  should  have  software  to  verify  the  operators’  decisions.  While  the 
operator  should  retain  control  and  authority  as  to  how  to  proceed,  disagreements  should  be  logged 
automatically  (EPRI,  1993a).  Alerting  crews  to  possible  unintentional  deviations  from  the  procedure  was 
identified  as  a  potential  improvement  to  the  crew’s  reliability  by  enabling  them  to  recover  from  mistakes 
(Orvis  and  Spurgin  ,1996;  Moieni  and  Spurgin,  1993b)  and  to  catch  “local”  errors  (Jeffroy  and  Charron, 
1997).  Other  studies  also  identified  this  as  a  desirable  CBP  feature  (NRC  CBP  Workshop).  By  alerting 
operators,  they  can  decide  if  that  is  what  they  want  to  do  (Pirus  and  Chambon,  1997).  This  guideline  is  an 
application  of  the  High-Level  Design  Review  Principles  of  Error  Tolerance  in  Control,  Feedback,  and 
Situation  Awareness  (see  Appendix  B).  However,  care  must  be  taken  to  assure  that  operators  are  not 
reluctant  to  deviate  from  the  CBP.  As  Jeffroy  and  Charron  (1997)  noted,  there  are  situations  where 
operators  may  disagree  with  the  CBP’s  recommendations  and  may  find  it  hard  to  disagree  with  the 
procedures,  especially  when  the  level  of  detail  in  the  CBP  is  high. 

10.2.4  Planning  and  Implementation 

10.2.4- 1  Display  of  Action  Status 

The  status  of  procedure-related  actions  should  be  displayed  by  the  CBP. 

Discussion:  This  feature  is  a  potential  improvement  to  crew  reliability  (Orvis  and  Spurgin,  1996).  This 
guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Situation  Awareness  and 
Feedback  (see  Appendix  B). 


NUREG/CR-6634 


10-  12 


10  CBP  HFE  DESIGN  REVIEW  GUIDELINES 


10.2.4-2  Timing  of  Procedures 

The  CBP’s  timing,  such  as  status  update  rates,  screen  changes,  and  navigation  features,  should  be 
consistent  with  the  time  demands  of  the  task. 

Discussion:  The  timing  of  CBP  responses  affects  operators’  performance.  Spurgin  et  al.  (1990)  indicated 
that  an  SRO  requested  ADS  initiation  twice  thinking  it  had  not  been  presented  after  the  first  request;  the 
misunderstanding  was  due  to  the  delay  in  the  CBPs  update  of  ADS  status  (Spurgin  et  al.,  1990).  This 
guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Timeliness  and  Feedback  (see 
Appendix  B). 

10.3  Management  and  Support  of  Procedures 

10.3.1  Path  Monitoring 

10.3.1- 1  Monitoring  Step  Status 

There  should  be  an  indication  of  whether  or  not  a  step  was  completed. 

ADDITIONAL  INFORMATION:  The  indication  can  be  manual  or  automatic,  depending  on  whether  the 
CBP  has  the  specific  criteria  and  information  to  determine  this. 

Discussion:  CBPs  can  keep  track  of  what  steps  have  been  completed,  using  check  boxes.  This  can  be 
manual  or  automatic,  depending  on  whether  the  CBP  has  the  specific  criteria  and  information  to 
determine  whether  a  step  was  completed.  Completion  also  can  be  time  stamped  to  facilitate  post-hoc 
incident  analysis  (Niwa,  Hollnagel,  and  Green,  1996).  This  guideline  is  an  application  of  the  High-Level 
Design  Review  Principles  of  Cognitive  Workload  and  Situation  Awareness  (see  Appendix  B). 

10.3.1- 2  Alert  User  to  Incomplete  Procedure  Steps 
Users  should  be  alerted  to  incomplete  procedure  steps. 

ADDITIONAL  INFORMATION:  The  alert  should  be  advisory  and  not  discourage  the  crew’s  actions. 
Discussion:  CBPs  should  monitor  whether  procedure  steps  were  not  fully  completed  and  notify  the  crew  if 
further  action  is  needed  (Orvis  and  Spurgin,  1996;  Moieni  and  Spurgin,  1993b).  This  guideline  is  an 
application  of  the  High-Level  Design  Review  Principles  of  Error  Tolerance  in  Control,  Feedback,  and 
Situation  Awareness  (see  Appendix  B). 

10.3.1- 3  Coding  Current  Location 

The  current  procedure  step(s)  should  be  indicated. 

Discussion:  Automatic  place  keeping  is  a  CBP  feature  that  can  improve  the  crew’s  reliability  (Orvis  and 
Spurgin,  1996),  especially  when  using  multiple  procedures.  This  guideline  is  an  application  of  the  High- 
Level  Design  Review  Principle  of  Cognitive  Workload  (see  Appendix  B). 

10.3.1- 4  Automatic  Path  Monitoring 

The  pathway  taken  through  procedures  should  be  stored  and  made  available  to  users. 

ADDITIONAL  INFORMATION:  A  history  should  be  maintained  and  available  for  display  on  request. 
Step  completion  can  be  time  stamped  to  facilitate  px)st-hoc  incident  analysis  (Niwa,  Hollnagel,  and  Green, 
1996). 

Discussion:  CBPs  can  keep  track  of  what  steps  have  been  completed;  this  can  be  manual  or  automatic 
depending  on  whether  the  CBP  has  the  specific  criteria  and  information  to  make  this  determination 
(Niwa,  Hollnagel,  and  Green,  1996).  This  guideline  is  an  application  of  the  High-Level  Design  Review 
Principle  of  Cognitive  Workload  (see  Appendix  B). 
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10.3.1- 5  Indication  of  Multiple  Active  Procedures 

The  user  should  be  informed  when  multiple  procedures  or  multiple  procedure  steps  are  to  be  followed 
concurrently.  A  list  of  all  currently  active  procedures  should  be  available. 

ADDITIONAL  INFORMATION:  It  may  be  helpful  for  the  list  of  active  procedures  to  include  start  and 
stop  times  for  the  procedures  in  use. 

Discussion:  See  discussion  for  Guideline  10.L2-1,  Concise  Steps.  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principle  of  Cognitive  Workload  (see  Appendix  B). 

10,3.2  Navigation 

10.3.2- I  Flexible  Navigation 

Navigation  support  should  allow  users  to  freely  and  easily  move  between  procedure  steps,  to  other  parts  of 
the  same  procedure,  and  to  other  procedures. 

ADDITIONAL  INFORMATION:  Users  should  not  be  forced  to  access  procedures  in  a  fixed  sequence  of 
the  procedure  nor  should  their  access  to  supporting  information  be  limited.  (See  also  the  additional 
information  on  Guideline  10.2.1-1.) 

Discussion:  Navigation  within  one  procedure,  or  among  multiple  procedures  and  related  supporting 
information,  can  be  time  consuming  and  error  prone  (O'Hara,  Stubler,  and  Higgins,  1 996).  Collier  (1996) 
noted  that  the  CBP  system  should  not  overly  structure  the  operator's  movement  through  the  procedure  but 
should  offer  flexibility  for  operators  to  skip  steps  or  skim  over  them  quickly.  Operators  need  to  move 
easily  between  procedures  and  support  information  (Niwa,  Hollnagel,  and  Green,  1996),  in  part  to  make 
up  for  the  insufficiencies  of  procedures  (Dien  et  al.,  1991).  In  addition,  flexibility  improves  the  crew’s 
reliability  (Orvis  and  Spurgin,  1996).  This  guideline  is  an  application  of  the  High-Level  Design  Review 
Principles  of  Response  Workload  and  Flexibility  (see  Appendix  B). 

10.3.2- 2  Support  Parallel  Access  to  Information 

The  CBP  should  have  the  ability  to  access  more  than  one  piece  of  information  at  once. 

Discussion:  Hoecker  et  al.  (1994)  and  Hoecker  and  Roth  (1996)  found  that  the  workload  associated  with 
CBPs  can  increase  when  the  operators  cannot  access  needed  information  in  parallel.  Similarly,  CBPs  can 
significantly  improve  the  crew’s  performance  in  comparison  with  PBPs  if  they  allow  simultaneous  access 
to  multiple  procedures  (Wilhelmsen  et  al.,  1992).  This  guideline  is  an  application  of  the  High-Level 
Design  Review  Principles  of  Cognitive  Workload  and  Response  Workload  (see  Appendix  B). 

10.3.2- 3  Navigational  Links  to  Related  Information 

Navigational  links  to  cross-referenced  information  and  to  notes,  cautions,  warnings,  reference  material, 
and  communication  and  help  facilities  should  be  provided. 

ADDITIONAL  INFORMATION*.  Techniques  such  as  hyperlinks  can  expedite  navigation  to  information 
material  cross-referenced  in  a  procedure  or  its  supporting  material. 

Discussion:  Navigation  to  and  from  cross-referenced  material  can  be  time  consuming,  distracting,  and 
error  prone.  Computer  support  for  these  transitions  can  reduce  the  workload  associated  with  these  tasks. 
This  guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Response  Workload  and 
Flexibility  (see  Appendix  B). 

10.3.2- 4  Access  to  Contingency  Actions 

Users  should  be  able  to  easily  access  appropriate  contingency  actions. 

Discussion:  See  discussion  for  Guideline  1 0. 1 .2- 1 ,  Concise  Steps. 
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10.3.3  Help 


10.3.3- 1  Explanation  Facilities 

CBPs  should  have  facilities  to  enable  the  user  to  determine  how  CBP  functions  are  performed. 
ADDITIONAL  INFORMATION:  When  CBPs  support  users’  decision  making,  such  as  offering  advice  on 
how  to  select  procedures,  analyze  step  logic  or  follow  procedure  paths,  users  should  be  able  to  query  the 
basis  for  the  advice.  Cooperative  dialogue  enables  the  user  to  better  understand  and  utilize  the  system. 
Discussion:  In  general,  COSSs  often  are  not  designed  to  be  sufficiently  observable.  That  is,  they  do  not 
clarify  their  reasoning  basis,  nor  have  adequate  communication  facilities  to  enable  operators  to  question 
and  verify  system  performance.  Guidance  may  be  given  without  sufficient  communication  about  what  led 
to  its  issuance,  what  parameters  were  analyzed,  and  what  sequence  of  reasoning  was  followed.  When  the 
reasoning  process  is  shown,  it  may  conflict  with  that  of  the  operators,  i.e.,  it  may  be  based  on  the 
designer’s  theoretical  understanding  and  not  the  operators’  practical  experience  (Dien  and  Montmayeul, 
1995;  IAEA,  1994;  Malin  et  al.,  1991a;  Roth,  Bennett,  and  Woods,  1987).  Explanations  of  the  rationale 
for  procedure  steps  have  been  identified  as  a  necessary  CBP  feature  (Niwa  et  al.,  1996).  This  guideline  is 
an  application  of  the  High-Level  Design  Review  Principle  of  User  Guidance  and  Support  (see 
Appendix  B). 

10.3.3- 2  Help  Facilities 

Help  for  performing  procedure  specified  activities  should  be  provided. 

Discussion:  Information  should  be  given  to  help  operators  carry  out  procedure  steps.  For  example,  a  help 
facility  could  provide  information  as  to  how  a  control  action  should  be  carried  out  (Niwa,  Hollnagel,  and 
Green,  1996).  This  guideline  is  an  application  of  the  High-Level  Design  Review  Principle  of  User 
Guidance  and  Support  (see  Appendix  B). 

10.3.3- 3  Note  Taking 

There  should  be  a  way  for  users  to  record  their  notes  and  comments  in  the  CBP. 

Discussion:  Procedures  have  gaps  because  they  do  not  cover  all  possible  situations  and  actions.  CBPs  can 
help  eliminate  them  by  allowing  operators  to  log  omissions  in  an  on-line  database  which  then  could  be 
accessed  to  identify  improvements  to  the  procedure  (NRC  CBP  Workshop).  This  guideline  is  an 
application  of  the  High-Level  Design  Review  Principle  of  Response  Workload  (see  Appendix  B). 

10.4  CBP  Hardware 

Guidance  for  CBP  hardware,  including  VDUs,  printers,  computer  input  devices,  is  part  of  the  CBP  review,  and  is 
available  in  NUREG-0700.  An  additional  consideration  is  discussed  below. 

10.4-1  Number  of  VDUs 

The  number  of  VDUs  on  which  CBP  information  is  displayed  should  be  sufficient  to  provide  all  the 
procedure- related  information  needed  for  a  procedure  step,  including  cautions  and  reference  material. 
Discussion:  VDUs  can  create  a  keyhole  effect  and  the  requirement  for  potentially  distracting  interface 
management  tasks.  This  guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of 
Situation  Awareness  and  Task  Compatibility  (see  Appendix  B). 
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10.5  Backup  for  CBP  Procedures 

10.5- 1  Paper-Based  Procedure  Availability 
PBPs  should  be  available  in  the  event  of  CBP  failure. 

Discussion:  PBPs  will  enable  operators  to  perform  safety-related  tasks  in  situations  where  the  CBP  system 
is  malfunctioning  or  has  failed.  This  guideline  is  an  application  of  the  High-Level  Design  Review 
Principle  of  Error  Tolerance  and  Control  (see  Appendix  B). 

1 0.5- 2  Consistency  of  PBPs  and  CBPs 

The  content  and  presentation  of  procedure  information  in  PBPs  and  CBPs  should  be  consistent. 
ADDITIONAL  INFORMATION:  Smooth  transfer  between  CBPs  and  PBPs  and  vice  versa  will  be 
facilitated  by  the  degree  to  which  their  formatting  is  consistent;  this  also  will  facilitate  training  in 
procedure  use. 

Discussion:  The  hard-copy  procedures  should  be  consistent  in  format  and  content  with  the  CBPs.  EPRl 
noted  that  their  consistency  will  minimize  the  training  burden  and  lower  the  potential  for  errors  and 
misunderstandings.  This  consideration  is  especially  important  when  the  hard-copy  procedures  have  to  be 
used  as  a  backup  (EPRI,  1993a).  This  guideline  is  an  application  of  the  High-Level  Design  Review 
Principles  of  Consistency  and  Error  Tolerance  and  Control  (see  Appendix  B). 

10.5- 3  Support  for  Transfer  to  PBPs 

Upon  transfer  to  PBPs,  a  means  should  be  provided  to  support  the  user’s  determination  of  currently  open 
procedures,  location  in  the  procedures,  completed  and  not  completed  steps,  and  currently  monitored  steps. 
ADDITIONAL  INFORMATION:  When  the  CBP  is  lost,  it  may  be  difficult  for  operators  to  reconstruct 
this  information  from  memory.  Therefore,  the  operator  should  be  supported  in  making  a  safe,  easy 
transition.  For  example,  a  CBP  system  might  automatically  print  out  a  status  sheet  with  this  information 
once  every  minute  so  that  if  it  fails,  the  operator  can  retrieve  the  latest  sheet  and  use  it  to  establish  the 
crew’s  tasks  for  using  PBPs. 

Discussion:  Operators  may  be  in  multiple  procedures  when  the  CBP  fails.  For  each,  the  CBP  may  have 
been  monitoring  progress,  monitoring  and  evaluating  steps  of  continuous  applicability  and  other  steps  in 
the  background.  Providing  these  supports  is  one  of  the  CBP  benefits  that  reduce  the  operators’  cognitive 
workload  in  remembering  this  information.  When  the  CBP  is  lost,  it  may  be  difficult  for  operators  to 
reconstruct  this  information  from  memory.  Therefore,  some  means  should  be  provided  to  support  this 
transfer.  This  guideline  is  an  application  of  the  High-Level  Design  Review  Principles  of  Cognitive 
Workload  and  Error  Tolerance  and  Control  (see  Appendix  B). 

10.6  Integration  of  CBPs  with  Other  HSI  Components 

10.6- 1  Consistency  with  Other  HSI  Conventions 

The  detailed  CBP  design  should  be  fully  consistent  with  the  rest  of  the  HSI. 

ADDITIONAL  INFORMATION:  HSI  features  for  format  and  functionality  (such  as  labeling,  acronyms, 
dialog  conventions,  use  of  colors,  and  input  devices)  should  be  consistent  between  the  CBP  and  other  HSI 
components.  Consistency  may  be  a  special  consideration  when  reviewing  “off-the-shelf’  systems. 
Discussion:  Lack  of  consistency  between  CBPs  and  the  other  HSI  resources  was  identified  as  an 
important  consideration  (NRC  CBP  Workshop).  Any  such  inconsistency  can  degrade  the  operator’s 
performance  and  increase  the  likelihood  of  errors.  Thus,  inconsistency  was  identified  as  a  potential  source 
of  risk  and  reduced  performance  reliability  (Niwa  et  al.,  1996).  This  guideline  is  an  application  of  the 
High-Level  Design  Review  Principle  of  Consistency  (see  Appendix  B). 
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A.l  Background 

In  the  nuclear  industry,  the  development  of  procedures  historically  was  considered  the  responsibility  of  individual 
utilities,  but  the  rationale  for  including  a  procedure  development  element  in  NUREG-071 1  is  that  procedures  are 
an  essential  component  of  the  HSl  design,  and  should  be  derived  from  the  same  design  processes  and  analyses  as 
other  HSIs  (e.g.,  displays,  controls,  operator  aids)  and  evaluated  in  the  same  way.  Technically  detailed,  emergency 
operating  procedures  (EOPs)  were  an  improvement  instituted  after  the  accident  at  Three  Mile  Island  (TMI)  to 
support  safe  operations.  First,  the  NPP  owners  groups  developed  generic  technical  guidance  (GTG)  and  utilities 
then  produced  EOPs  based  on  the  GTG.  Thus,  procedure  development  programs  were  conducted  by  the  individual 
utilities  and  were  not  part  of  HSl  design  activities.  However,  since  procedures  were  developed  after  the  design  of 
the  plant  HSl  (e.g.,  control  room),  they  were  essentially  retrofitted  to  suit  the  existing  interface.  Further,  since  they 
were  established  by  individual  utilities,  their  development  and  final  implementation  varied  greatly.  As  a  result, 
human  factors  problems  existed,  and  the  identification,  access,  interpretation,  and  validation  of  procedures 
remained  troublesome  for  years  in  several  plants,  as  shown  by  the  NRC  EOP  inspection  series  (Lapinsky,  1989; 
Galletti  and  Sutthoff,  1 992).  In  addition,  inconsistencies  between  procedures  and  the  HSl  have  been  a  source  of 
difficulty  for  operators. 

For  new  plant  designs  and  advanced  reactors,  these  problems  should  clearly  be  addressed  and  solved  during  the 
design  process.  To  accomplish  this  objective,  GTG  and,  if  possible,  procedures  should  be  developed  as  part  of  the 
same  design  process  as  that  for  other  components  of  the  HSl  to  ensure  their  full  integration  into  the  HSl.  The  same 
human  factors  analyses,  such  as  task  analysis,  should  be  used  to  guide  the  design  of  the  control  panel,  as  well  as 
procedure  development.  The  same  human  factors  principles  should  be  applied  to  both  aspects  of  the  interface  to 
ensure  complete  integration  and  consistency.  Further,  procedures  should  be  evaluated  in  conjunction  with  the  HSl; 
procedures  are  a  significant  aspect  of  system  verification  and  validation  (Element  10). 

A.2  Objective 

The  objective  of  this  review  is  to  ensure  that  the  applicant’s  procedure  development  program  will  result  in 
procedures  that  guide  human  interactions  with  plant  systems  and  control  plant-related  events.  Human  engineering 
principles  and  criteria  should  be  applied,  along  with  all  other  design  requirements,  to  develop  technically  accurate, 
comprehensive,  explicit,  easy  to  utilize,  and  validated  procedures. 

A.3  Applicant  Submittals 

The  applicant  should  provide  the  following  documents  for  staff  review:  implementation  plan,  analysis-results 
report,  and  HFE  design  team  evaluation  report.  Section  1.4.4  (of  NUREG-071 1)  describes  these  submittals. 

In  addition,  GTG  and  draft  procedures  should  be  available  for  review. 

A.4  Review  Criteria 

(1)  The  following  procedures  are  within  the  scope  of  the  element: 

•  GTG  for  EOPs 
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•  Plant  and  system  operations  (including  startup,  power,  and  shutdown  operations) 

•  Abnormal  and  emergency  operations 

•  Preoperational,  startup,  and  surveillance  tests 

•  Alarm  response* 

(2)  The  basis  for  procedure  development  should  include: 

•  Plant  design  bases 

•  System-based  technical  requirements  and  specifications 

•  Results  of  task  analyses 

•  Risk- important  human  actions  identified  in  the  HRA/PRA 

•  Initiating  events  to  be  considered  in  the  EOPs,  including  those  events  in  the  design  bases 

•  GTG  for  EOPs 

(3)  A  writer’s  guide  should  be  developed  to  establish  the  process  for  developing  technical  procedures  that  are 
complete,  accurate,  consistent,  and  easy  to  understand  and  follow.  The  guide  should  contain  sufficiently 
objective  criteria  so  that  resulting  procedures  are  consistent  in  their  organization,  style,  and  content.  The 
guide  should  be  used  for  all  procedures  within  the  scope  of  this  element.  It  should  provide  instructions  on 
the  procedures’  content  and  format,  including  writing  action  steps  and  specifying  acceptable  acronym  lists 
and  terms. 

(4)  The  content  of  the  procedures  should  incorporate  the  following  elements: 

•  Title 

•  Statement  of  applicability 

•  References 

•  Prerequisites 

•  Precautions  (including  warnings,  cautions,  and  notes) 

•  Limitations  and  actions 

•  Required  human  actions 

•  Acceptance  criteria 

•  Checkoff  lists 
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(5)  In  addition  to  the  general  procedure  elements  identified  in  Criterion  4  above,  GTG  should  be  symptom 
based  with  clearly  specified  entry  conditions. 

(6)  All  procedures  should  be  verified  and  validated;  a  review  should  ensure  they  are  correct  and  can  be  carried 
out.  They  should  be  finally  validated  in  a  simulation  of  the  integrated  system,  as  part  of  the  verification 
and  validation  activities  described  in  Element  10. 

(7)  An  analysis  should  determine  the  impact  of  providing  computer-based  procedures,  CBPs,  (either  p^irtial  or 
complete)  and  specify  where  such  an  approach  would  improve  the  use  of  procedures  and  reduce  related 
errors  by  the  operating  crew.  Justification  for  using  CBPs  rather  than  paper  procedures  should  be  given. 
An  analysis  should  be  made  and  documented  of  alternatives  in  the  event  of  loss  of  CBPs. 

(8)  A  plan  for  maintaining  procedures  and  controlling  updates  should  be  developed. 

(9)  The  physical  means  by  which  operators  access  and  use  procedures,  especially  during  operational  events, 
should  be  evaluated  as  part  of  the  HFE  design  process.  This  criterion  generally  applies  to  both  hard-copy 
and  computer-based  procedures,  although  the  types  of  issues  differ  somewhat  for  them.  For  example,  the 
process  should  address  the  storage  of  procedures,  ease  of  operator  access  to  the  correct  procedures,  and 
laydown  of  hard-copy  procedures  for  use  in  the  control  room,  remote  shutdown  facility,  and  local  control 
stations. 

(10)  The  following  documents  may  be  used  as  guidance  (per  Section  1.4.4): 

NUREG-0800,  Rev.  1:  Standard  Review  Plan,  1984  (NRC). 

NUREG-0899:  Guidelines  for  the  Preparation  of  Emergency  Operating  Procedures,  1 982  (NRC). 

NUREG-1358:  Lessons  Learned  From  the  Special  Inspection  Program  for  Emergency  Operating 
Procedures,  1 989  (NRC). 

NUREG-1 358:  Lessons  Learned  From  the  Special  Inspection  Program  for  Emergency  Operating 
Procedures,  Supplement  1,  1989  (NRC). 

NUREG/CR-5228:  Techniques  for  Preparing  Flowchart  Format  Emergency  Operating  Procedures, 
Volumes  1  and  2,  1989  (NRC  -  Barnes  et  al.). 

NRC  Regulatory  Guide  1.33,  Rev.  2:  Quality  Assurance  Program  Requirements,  1978  (NRC). 

ANS  3.2-1994:  Administrative  Controls  and  Quality  Assurance  for  the  Operational  Phase  ofNPPs,  1994 
(American  Nuclear  Society). 

BNL  TR  E2090-T4-2-9/96:  Preliminary  Review  Criteria  for  Evaluating  Computer-Based  Procedures, 
1996  (Barnes  et  al.) 


A-3 


NUREG/CR-6634 


APPENDIX  B 

High-Level  Design  Review  Principles  from 
NUREG-0700,  Rev.  1 


NUREG/CR-6634 


HIGH-LEVEL  DESIGN  REVIEW  PRINCIPLES  FROM  NUREG-0700 


The  design  of  human-system  interfaces  (HSIs)  should  support  the  operating  personnel’s  primary  task  of  monitoring 
and  controlling  the  plant,  without  imposing  an  excessive  workload  associated  with  using  the  HSI  (manipulating 
windows,  selecting  displays,  and  navigating,  for  example).  The  HSI  also  should  support  the  recognition,  tolerance, 
and  recovery  from  any  human  errors.  Guidelines  for  reviewing  human  factors  engineering  designs  help  to  ensure 
that  these  goals  are  achieved.  As  part  of  the  guidance  development  for  NUREG-0700,  Rev.  1 ,  a  set  of  “high-level” 
design  review  principles  was  developed  representing  the  generic  HSI  characteristics  necessary  to  support  personnel 
performance.  They  were  used  to  develop  many  detailed  review  guidelines  in  Part  2  NUREG-0700  (O'Hara,  Brown, 
and  Nasta,  1996  discuss  their  use).  The  high-level  principles  also  were  used  in  formulating  guidelines  for 
computer-based  procedures. 

The  1 8  principles  are  divided  into  four  categories:  general  principles,  primary  task  design,  secondary  task  control, 
and  task  support.  The  categories  and  the  principles  that  underlie  them  are  described  below. 

B.l  General  Principles 

These  principles  ensure  that  the  HSI  design  supports  personnel  safety,  and  is  compatible  with  their  general 
cognitive  and  physiological  capabilities. 

•  Personnel  Safety  -  The  design  should  minimize  the  potential  for  injury  and  exposure  to  harmful  materials. 

•  Cognitive  Compatibility  -Ttit  operators'  roles  should  consist  of  purposeful,  meaningful  tasks  that  enable  them 
to  maintain  familiarity  with  the  plant  and  maintain  a  level  of  workload  that  is  not  so  high  as  to  lower 
performance,  but  sufficient  to  maintain  vigilance. 

•  Physiological  Compatibility  -Vtit  design  of  the  interface  should  reflect  consideration  of  human  physiological 
characteristics,  including  visual/auditory  perception,  biomechanics  (reach  and  motion),  characteristics  of 
motor  control,  and  anthropometry. 

•  Simplicity  of  Design  -  The  HSI  should  represent  the  simplest  design  consistent  with  function  and  task 
requirements. 

•  Consistency  -  There  should  be  a  high  degree  of  consistency  between  the  HSI,  the  procedures,  and  the  training 
systems.  At  the  HSI,  the  way  the  system  functions  and  appears  to  the  operating  crew  always  should  be 
consistent,  reflect  a  high  degree  of  standardization,  and  be  fully  consistent  with  procedures  and  training. 

B.2  Primary  Task  Design 

These  principles  support  the  operator's  primary  tasks  of  monitoring  and  detection,  situation  assessment,  response 
planning,  and  response  implementation. 

•  Situation  Awareness  -  The  information  presented  to  the  users  by  the  HSI  should  be  correct,  rapidly  recognized, 
and  easily  understood  (e.g.,  “direct  perception”  or  “status  at  a  glance”  displays)  and  support  the  higher-level 
goal  of  their  awareness  of  the  system’s  status. 

•  Task  Compatibility  -  The  system  should  meet  the  requirements  of  users  in  performing  their  tasks  (including 
operation,  safe  shutdown,  inspection,  maintenance,  and  repair).  Data  should  be  presented  in  forms  and 
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formats  appropriate  to  the  task  (including  the  need  to  access  confirmatory  data  or  raw  data  in  the  case  of 
higher-level  displays),  and  control  options  should  encompass  the  range  of  potential  actions.  There  should  be 
no  unnecessary  information  or  control  options. 

•  User  Model  Compatibility  -  All  aspects  of  the  system  should  be  consistent  with  the  users'  mental  models 
(understanding  and  expectations  about  how  the  system  behaves  from  training,  use  of  procedures,  and 
experience).  All  aspects  of  the  system  also  should  be  consistent  with  established  conventions  (i.e.,  expressed 
in  customary,  commonplace,  useful  and  functional  terms,  rather  than  abstract,  unusual  or  arbitrary  forms,  or  in 
forms  requiring  interpretation). 

•  Organization  of  HSI  Elements  -  The  organization  of  all  aspects  of  the  HSI  (from  the  elements  in  individual 
displays,  to  individual  workstations,  to  the  entire  control  room)  should  be  based  on  the  users’  requirements  and 
should  reflect  the  general  principles  of  organization  by  importance,  frequency,  and  order  of  use.  Critical  safety 
function  information  should  be  available  to  the  entire  operating  crew  in  dedicated  locations  to  ensure  its 
recognition,  and  to  minimize  data  search  and  response. 

•  Logical/Explicit  Structure  -  All  aspects  of  the  system  (formats,  terminology,  sequencing,  grouping,  and 
operator's  decision-support  aids)  should  reflect  an  obvious  logic  based  on  task  requirements  or  some  other  non- 
arbitrary  rationale.  The  relationship  of  each  display,  control,  and  data-processing  aid  to  the  overall  task  and 
function  should  be  clear.  The  structure  of  the  interface  and  its  associated  navigation  aids  should  make  it  easy 
for  users  to  recognize  where  they  are  in  the  data  space,  and  should  enable  them  to  rapidly  access  data  not 
currently  visible  (e.g.,  on  other  display  pages).  The  way  the  system  works,  and  is  structured,  should  be  clear  to 
the  user. 

•  Timeliness  —  The  system’s  design  should  take  into  account  users'  cognitive  processing  capabilities  as  well  as 
process-related  time  constraints  to  ensure  that  tasks  can  be  performed  within  the  required  time.  Information 
flow  rates  and  control  performance  requirements  that  are  too  fast  or  too  slow  could  diminish  performance. 

•  Controls/Displays  Compatibility  -  Displays  should  be  compatible  with  the  requirements  for  data  entry  and 
control. 

•  Feedback -Tht  system  should  provide  useful  information  on  its  status,  permissible  operations,  errors  and 
error  recovery,  dangerous  operations,  and  validity  of  data. 

B.3  Secondary  Task  Control 

These  principles  minimize  secondary  tasks,  i.e.,  tasks  personnel  must  perform  when  interfacing  with  the  system 

that  are  not  directed  to  the  primary  one.  Examples  include  managing  the  interface,  such  as  navigation  through 

displays,  manipulating  windows,  and  accessing  data.  Performing  secondary  tasks  detracts  from  the  crew's  primary 

tasks,  so  the  demands  of  secondary  tasks  must  be  controlled. 

•  Cognitive  Workload  -  The  information  presented  by  the  system  should  be  rapidly  recognized  and  understood; 
therefore,  the  system  should  minimize  requirements  for  making  mental  calculations  or  transformations  and 
using  recall  memory  (recalling  lengthy  lists  of  codes,  complex  command  strings,  information  from  one  display 
to  another,  or  lengthy  action  sequences).  Raw  data  should  be  processed  and  presented  in  directly  usable  form 
(although  raw  data  should  be  accessible  for  confirmation). 
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•  Response  Workload  -  The  system  should  require  a  minimum  number  of  steps  to  accomplish  an  action;  e.g., 
single  versus  command  keying,  menu  selection  versus  multiple  command  entry,  single  input  mode  (keyboard, 
mouse)  versus  mixed  mode.  In  addition,  the  system  should  not  require  redundant  data  to  be  entered,  nor  the 
re-entry  of  information  already  in  the  system,  or  information  the  system  can  generate  from  already  resident 
data. 

B.4  Task  Support 

These  principles  address  the  characteristics  of  the  HSl  that  support  its  use  by  personnel,  such  as  providing  (1)  HSl 

flexibility  so  tasks  can  be  accomplished  in  more  than  one  way,  (2)  guidance  for  users,  and  (3)  mitigation  of  errors. 

•  Flexibility  -  The  system  should  give  the  user  multiple  means  to  carry  out  actions  (and  verify  automatic 
actions)  and  permit  displays  and  controls  to  be  formatted  in  a  configuration  most  convenient  for  the  task. 
However,  flexibility  should  be  limited  to  situations  where  it  is  advantageous  for  task  performance  (such  as  to 
accommodate  different  levels  of  experience  of  the  users);  it  should  not  be  provided  for  its  own  sake  because 
there  is  a  tradeoff  between  flexibility  and  the  increase  in  interface  management  workload  (which  detracts  from 
monitoring  and  operations). 

•  User  Guidance  and  Support  -  The  system  should  provide  an  effective  “help”  function.  Informative,  easy-to- 
use,  and  relevant  guidance  should  be  given  on-line  and  off-line  to  help  the  user  understand  and  operate  the 
system. 

•  Error  Tolerance  and  Control  -  A  fail-safe  design  should  be  provided  wherever  failure  can  damage  equipment, 
injure  personnel,  or  inadvertently  operate  critical  equipment.  Therefore,  the  system  should  generally  be 
designed  such  that  a  user’s  error  will  not  have  serious  consequences.  The  negative  effects  of  errors  should  be 
controlled  and  minimized.  The  system  should  offer  simple,  comprehensible  notification  of  the  error,  and 
simple,  effective  methods  for  recovery. 
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